The power adapter (old and new, as well as iBook and PowerBook adapters) contains a "switching power supply." This sort of power supply is both compact and lightweight, and is highly efficient. It is not uncommon for switching power supplies that are under load to emit an audible hum or buzz; this is merely the switching operation causing a physical vibration. You may have noticed a lot of photoflashes emit a "whine" while charging-this is the same phenomenon.
Now as to whether it's a problem or not-that depends. If the adapter still charges the battery in the same sort of timeframe, if it does not get noticeably hotter, and if it's sound remains relatively quiet, I'd say it's ok (if annoying). If it does any of the above things though, I'd contact Apple right away.