Isn't there an old joke about this:
"Doctor, my computer reboots when I do this."
"So don't do that!"
Seriously, there could be a loose connection somewhere, most obviously the power plug into the back (assuming this is an iMac G5 or Intel). If you hold the machine when you do this instead of letting go, does it stay shut off? Of course you can also try to usual (reset PRAM, NVRAM, repair permissions, etc.). If you do a little searching, I seem to recall some iMac G5 logic board recall at some point. Others may know more about it.
Steve