I have experienced the same problem.
Using a Powerbook 17" with a M-Audio Sonica USB. Connected using Toslink to a 5.1 Receiver.
I think you will find if you reboot with the USB audio device plugged in, the sound quality will be ok. But if you boot your machine, and then plug-in the device, the sound will be muffled like you describe.
Also I think sleeping the machine seems to affect it some how, but I have not worked out the exact pattern yet.
I was about to e-mail M-Audio about this, but now that I have seen someone else using a different device, I believe the problem may in fact be with OS X (or iTunes?)
This has been a problem with 10.2.4, 2.5, 2.6 (Used iTunes 4.0 and 4.01 across these versions). Have not tested with any Panther builds.
Perhaps someone with an M-Audio USB device and a copy of Panther could test this and let us know?