Ummmm... from my understanding of video sync modes, your iMac's video circuitry will autosense the resolutions and refresh rates supported by your monitor. You can try holding down control for the monitor res. control strip, or setting the monitors control panel to show all. if it doesn't list 1152x870, you can't do it. You'd likely blow the monitor if you could do it.
In other words: It's likely a hardware thing. Poking around the OS will do little good.