ok, so i am finally getting somewhere with this and i think my findings might help some of you with the same trouble.
first, you are able to tweak samba's send/receive buffers using the following parameters under you global settings in your smb.conf:
the two parameters are SO_RCVBUF/SNDBUF. i currently have mine set to 8mb or 8192 like above. this should work for most configurations. if you would like to refine your buffers, visit this site and look for socket options:
socket options = SO_RCVBUF=8192 SO_SNDBUF=8192
secondly you will also have to change your buffer/cache size in boxee. to do this you will have to use a terminal or putty in windows. in boxee this is found in the guisettings.xml file under:
to get there open up your terminal or putty and type in the following:
ok, now you are in the right directory. open up guisettings.xml using nano or other text editor and scroll down until you see this:
cd /mnt/Scratch/Users/frontrow/Library/Application Support/BOXEE/UserData/
now change <lan>2048<lan> to <lan>8192<lan>. now save the changes reboot your atv and restart your samba server. now with any luck you will be able to stream your movies from you samba share with no stuttering.
just to let you know i do get a stutter about 2 mins into a movie but that is the only time it does it now. i can live with one stutter for now. i hope this helps.