The system is probably splitting the graphics memory between the "two" monitors, hence the slowdown. Which PowerMac is it, and what graphics card(s) does it have? Better list any other cards installed too, in case one qualifies somehow.
If a display adapter is plugged into an extra display port, it will be read as a monitor even if one isn't connected. What does the system profiler say about each monitor? In particular, what info does it give for the phantom monitor?