It all depends on the codec as I'm sure k2director will be able to explain in far more detail.
Essentially, when you export a Quicktime movie, Quicktime is just sort of the wrapper around the movie, not the actual format, per se. The format, or the variable that dictates export times, is really the codec.
So if you're using Sorenson as your codec when exporting a Quicktime movie, this requires a lot of work when exporting so the movie's efficient when being played so it's going to take a long time to compress. Conversely, MPEG4, as I understand it, takes a much shorter time to compress because that's simply the nature of the codec.
I don't know if any of this helps. I'm going to post it anyway.