I think recent ffmpegX can play h.264? That's what I'm hoping because after you posted I downloaded the new handbrake, and am trying it out on a VIDEO_TS folder.
For the most part I don't find compressing movies particularly worth it, since it takes forever on my TiBook667, and disk space is pretty cheap.