I am willing to bet that the noise is from the CPU fan. It can be really annoying.
Listen to a recording of the 3 fans here:
http://scienceman.com/
The all in one design means that the fans are at eye/ear level which is bad by itself, additionally if you place your imac up against a wall, not at all uncommon, the fans are going to be even more noticable bec the noise echos off the wall and back at your head.
When apple announced the g5 imac i believe apple said it was even quieter than the previous gen imac. i find that very hard to believe. the g4 imac was the quietest computer i have ever owned, sometimes i honestly could not tell it was on or whether it was asleep. If i recall correctly the g4 imac was fanless at least on the cpu. the imac g5 was a real design challenge bec the g5 generates so much more heat than the g4. to deal with the heat apple's approach has been to utilize multiple smart fans that throttle up and down. it works pretty well but its not perfect. even at idle a cpu fan is often quite bothersome.