Transcoding is more CPU-intensive than gaming; the reason you don't have this problem when gaming is that it's the GPU, not the CPU, that is being stressed the most when gaming. In particular, most games can only take full advantage of one or two cores, whereas transcoding can use all of your cores.
Make sure all the case fans are working properly and are clear of dust. Also, make sure there's adequate ventilation around the computer. If that all checks out, you should re-seat the CPU's heat sink (and apply new thermal paste), as it may have come loose or have inadequate/too much thermal paste applied to it.
To test to make sure your cooling is adequate, you should download and run Prime95, start the "Torture Test" with 8 threads, and let it run overnight. Then download FurMark, which will test the GPU in the same way. Any computer that overheats while running either program does NOT have sufficient cooling. Adequate cooling means that no software should ever be able to cause your computer to overheat; these two programs are designed to make your CPU and GPU as hot as possible to ensure your cooling is adequate.