This guide will be focusing on AVC (h.264) encoding using the x264 encoder. This codec and encoder will produce the best quality for a given file size compared to alternatives. The x264 encoder is both free and superior to all of the commercial h.264 encoders out there. The Boxee Box depends upon hardware acceleration, so this guide will focus on preserving as much quality from the source media as possible without sacrificing compatibility.
If you use my presets, your resulting files will almost certainly NOT be compatible with a mobile device like an iPod or smart phone and may or may not work with game consoles. These presets are for modern, hardware-accelerated HTPCs and media tanks like WMC, XBMC, or Boxee. H.264 .MKV/.MP4 files MUST be supported by your device. There is no single setting or even handful of settings that will produce the best quality regardless of the source material or the application it's intended for. The settings I'm using here should be considered general suggestions. Feel free to experiment.
Also, I will be assuming that whatever source media you're using (DVD or BluRay) has already been decrypted. I'd suggest DVDDecrypter, AnyDVD, or DVDFab. They all have their own advantages and disadvantages. But I won't be discussing this in-depth because it just adds "noise" to the topic I want to focus on and the legality of some of this software is in dispute in some countries.
To preserve my sanity, I'll be focusing on two different automated software suites that use the x264 encoder, and I'll be addressing them separately. They will both accept a wide variety of source media and they will both produce excellent results. Make sure you read the front page and the FAQ for both of these software suites before you attempt their use for the first time. Pay special attention to the required software section for HDConvertToX. I will NOT be reprinting their FAQ sections in this guide:
Handbrake (for Windows) (With experimental BluRay support) Handbrake FAQ/Guide
HDConvertToX (Works with unencrypted BluRay) HDConvertToX Manual
Handbrake (Ver. 0.9.5)
1. Choose your unencrypted source material, be it a BluRay, DVD or a media file with an extension like .avi, .mkv, .vob, etc. Handbrake will then scan the file and parse out its video streams, audio streams, and subtitles. Then set your destination and file name in the "Destination" field. Handbrake will automatically select your entire movie for this project. If you want to select individual chapters and/or viewing angles, you'll have to do it manually.
2. Select your preset in the panel on the right. To make things MUCH easier, I'm including my Handbrake presets in this post. Download these presets here and import them by opening Handbrake, choosing "Presets" at the top, and then selecting "Import". Linux presets available at the bottom of this guide. From here on out, the guide will assume that you have done this.
You can start from scratch with one of the built-in presets or you can choose a "Prospero" preset. The "Prospero" presets already have most settings populated for maximum quality and Boxee/DXVA compatibility. If your source is interlaced (say, a 1080i file from a P/DVR) and you want your encode to result in a 720p .mkv, for example, choose "Prospero Interlaced Source 720p Encode". If your source is an American DVD rip, choose "Prospero NTSC DVD Source". If your source is a European DVD rip, choose "Prospero PAL DVD Source". If your source is a BluRay rip or a standard media file (.avi, .mkv, .mov, etc.), use one of the "Prospero Progressive Source..." Presets.
If you use one of my presets, it will automatically select "MKV File" as the container. If not, it will default to "MP4 File". Either should actually work with my presets aside from audio format selection, which you should keep an eye on in any case.
Now we'll move on to the "Picture" tab in the lower half of the Handbrake program window. Normally, you'll want to leave the "keep aspect ratio" box checked for three-letter extension video files, but different DVDs can be special cases. Rather than clutter up this guide, here's an explanation of the different settings. My presets should handle this automatically in most cases as long as you choose the correct preset for your source. The exception is the resolution settings, which now cannot be populated properly with presets. If you want your encode to result in a 1080p file, set the width to 1920 and just click on the height field to update it. If you want your encode to result in a 720p file, set it to 1280. If you want your encode to result in a 480p (standard def) file, set it to 720.
Now for the "Video Filters" tab. Unfortunately, there's no one catch-all setting I can suggest. "Detelecine Default" should be safe if your source is a DVD. You can usually leave the filter settings at "Off" if you're converting from a standard media file (.avi, etc.), but even this isn't a sure thing. Here's a good guide for determining the proper settings to use. Again, my presets should handle this automatically in most cases as long as you choose the correct preset for your source.
Now the "Video" tab. You'll want to leave the "Video Codec" setting at "H.264 (x264)". For "Frame Rate (FPS)", you'll want to leave it at "same as source" unless you have specific needs.
Under "Quality", you'll have several options. If you want to hit a specific file size with your encode, you can use either the "Target Size (MB)" option or the "Average Bitrate (kbps)" option, and you'll want to check both the "2-Pass encoding" and the "Turbo first pass" boxes. I suggest anywhere from 900-1800kbps for a 480p (SD) encode, 3500-5500kbps for a 720p encode, and anywhere from 8000 to 15000kbps for 1080p.
If you need a faster encode (single pass) or want to hit a certain quality rather than a certain file size, you can use the "Constant Quality" slider instead of the above options. In this case, I suggest an "RF" value of somewhere between 17 and 23, with 17 giving you the largest file size and the highest quality and 23 giving you the lowest file size and quality. I use 18, personally.
The hardcore guys will first do a quality-based encode for each source they want to compress, and then use the average bitrate achieved in this encode to do another 2-pass bitrate-based encode. They then keep the bitrate-based encode while discarding the quality-based encode. This will ensure that you hit a minimum quality level and get the most quality for the size of your eventual file, but for most people that's overkill.
My presets specify a quality-based encode with an RF value of "20" for a good balance of speed and quality. For other encoding options, you'll have to enter them manually. If you want absolute maximum quality for the file size, you'll want to use 2-pass.
Now the "audio" tab. Again, there are a lot of options here, but I suggest leaving the default options unless you know what you're doing. AC3 or DTS pass-through will give you the original surround sound, but you may wind up with a file whose audio stream is actually larger than its video stream if you're doing a low-bitrate SD encode. If this is the case, I suggest sticking with the stereo "AAC (faac)" or "MP3 (lame)" options. Most other options should be fine at their defaults. Consult the Handbrake wiki and forums for more info. My presets will automatically choose AC3 passthrough. If you have other needs, you'll need to enter them manually.
Handbrake should automatically populate the "Subtitles" and "Chapters" tabs, so I'm going to skip over them. But if your source has multiple languages included, make sure that you double-check the audio and subtitle streams Handbrake chooses.
The "Advanded" tab is complicated. I highly suggest using my attached presets. Going through each and every setting isn't feasible here. For a more thorough overview of the individual settings in the "Advanced" tab, check out this previous post and this x264 settings guide or the Handbrake wiki and forums.
That's it! You're ready to hit the "Start" button. This WILL be slow because you're really squeezing as much quality as possible out of it. My encodes usually take anywhere from 1-18 hours on an Intel Core 2 Quad Q9300 (2.5Ghz), depending upon the format, length, and resolution of the input and output files. The good thing is that x264 scales REALLY well when it comes to thread prioritization. Even with all four cores pegged at 100%, I'm still able to play all of my games, watch all of my media, etc. So keep that in mind: as long as your system is stable, it's only a matter of patience.
If your encode looks too compressed (blocky and fuzzy), either raise the bitrate/filesize or lower the RF value and encode it again. If your encoded file is too large, do the opposite.