I'm not sure what fixed it, but the problem seems to be related to the 'imported' user profile from OS X 10.3.9.
When I set up the new iMac, I used the utility to 'import' the user from my old machine. For whatever reason, I could never get the network control panel to work correctly.
I created a new user and manually moved everything over. It still didn't work right, but at least the network control panel wouldn't crash everytime I opened it...
Finally, I created a new 'Location' the Network control panel. This fixed it.
So to recap, when I leave it on "automatic", download speeds are compromised. When I created a new 'location' with all the same settings as 'automatic' it now works correctly. Go figure.