I have the same problem. I have Toast Titanium 9.0.2 and a Sony 4X Blu-ray writer. When I attempt to write a Blu-ray disc with HD video, it starts encoding and says "Encoding 'video name'." However, after a few seconds it just sits there. It's using almost no CPU and doing nothing.
I'm using .m2t files which I exported from iMovie. I had the same problem attempting to write a Blu-ray HD video to a DVD-ROM but, after diddling and diddling and changing discs, it finally encoded and wrote. I was able to play the HD video on my Playstation 3 (but it was jumpy).
After diddling with my Blu-ray project and changing discs, it still won't encode. What is up with this? This software isn't doing what it is advertised to do.