Plug in a monitor that can handle a high refresh rate and then set it back to 800x600 -OR- ssh into the machine and try deleting /Library/Preferences/com.apple.windowserver.plist
Deleting that file worked in 10.2.. Haven't used it since, and it certainly won't hurt.
Other things to try...
Reset PRAM (CMD-OPT-P-R on boot)
SAFE BOOT (Hold down shift on boot)