Modern video codec playback is a threshold event, since most frames rely heavily on previous frames (hence the compression), so obviously if you barely miss at playing any particular frame, many of the following frames will also be missed. "Just that bad" may be a matter of a couple extra cycles.
What machine are you using, what QT WMV plugin, and are you running QT in Rosetta for any reason?