OK, this is pretty rough and will probably need updating for bits I've missed, but its what has worked for me (It assumes a moderately recent ubuntu system):
tar xJvf boxee-ce4100-17443-sources.tar.xz
sudo apt-get install python-support cmake autotools-dev autoconf automake unzip libboost-dev zip libtool libgl1-mesa-dev libgl-dev libglu1-mesa-dev libglu-dev libglew-dev libmad0-dev libjpeg-dev libsamplerate-dev libogg-dev libvorbis-dev libfreetype6-dev libfontconfig-dev libbz2-dev libfribidi-dev libsqlite3-dev libmysqlclient-dev libasound-dev libpng-dev libpcre3-dev, liblzo2-dev libcdio-dev libsdl-dev libsdl-image1.2-dev libsdl-mixer1.2-dev libenca-dev libjasper-dev libxt-dev libxtst-dev libxmu-dev libxinerama-dev libcurl-dev libdbus-1-dev libhal-storage-dev libhal-dev libpulse-dev libavahi-common-dev libavahi-client-dev libxrandr-dev, libavcodec-dev, libavformat-dev, libavutil-dev libpostproc-dev libswscale-dev libmpeg2-4-dev, libass-dev libmpcdec-dev libflac-dev libwavpack-dev python-dev gawk libcwiid1-dev libbluetooth-dev zlib1g-dev libsmbclient-dev libfaad-dev libiso9660-dev libssl-dev lsb-release libvdpau-dev libmicrohttpd-dev libmodplug-dev librtmp-dev libcrystalhd-dev curl libgl1-mesa-dev libvdpau-dev
(this should hopefully fix some of the problems we were having during compiling)
./configure --disable-ss-rsxs --disable-boxee-hal
8 There's almost certainly some more dependencies that I've missed, so let me know and I can get them added.
This might have problems on a VDPAU systemas mentioned by Silverlock, but I don't have one to test on unfortunately.