I'm taking a guess here...
A number of PowerMacs of that era had some sort of graphics accleration for onboard video, probably an ATI chip. There could be a conflict occuring between the extensions to support the VR card and those that are loaded when a monitor is sensed on the built-in video port.
Do you get the same problem when you boot the machine with extensions disabled? You won't benefit from the acceleration features, but you will at least be able to see whether your problem is caused by an extension conflict.
If the problem does go away with all extensions disabled, try disabling only thos extensions that have to do with built in video, perhaps "Performa Monitors" or something like that (I don't have one of these machines, so I can't be more specific).
You may lose the moderate graphics acceleration of the built in video support, but I imagine this isn't driving your primary (or even secondary) monitor, so you won't worry about the slower response.