Yeah, you really don't want to just dump hundreds or thousands of fonts into ~/Library/Fonts if you're not going to use them. It *will* slow things down, and cause possible problems if any of the fonts are corrupt.
Instead, I'd use Font Book (it's a bit improved w/ the recent 10.3.x updates). It's not quite there for professional font management, perhaps, but it works pretty well.
You can use Font Book to disable collections, and easily turn them on and off. I'd recommend disabling the fonts as you import them. You can then browse the fonts in Font Book, and turn each font (or collections of fonts) on, as you need them.
Just make sure you have the option "Disabling turns off all fonts in a collection" to really disable. Also note that FB dumps fonts into ~/Library/Fonts willy-nilly, so if you care about keeping your own organized copy, turn "Always copy when installing fonts" on as well.