Usually, you can cleanly uninstall an app the same way you installed it. If you dragged it onto the hard drive, you can simply drag it to the Trash. If you ran an installer, then that same installer should allow you uninstall the program. Ironically, the only installer which cannot do this is Apple's own installer technology, but you can get around this using our own CharlesS' excellent Pacifist software.
Whoever told you that Preferences files can slow down your system is mistaken. This misconception probably comes from the fact that stray preferences can do this in Windows. This is because Windows stores most of its preferences in one large database called the Registry. This is one large database which says loaded into your system at all times. This provides many benefits, because preferences are easy to store and load. However, as the database gets bigger, your system slows down. Furthermore, since most computer users do not run all their applications at once, there is a significant waste of resources, because preferences which are not needed stay loaded anyway.
Macs, on the other hand, store their preferences in individual files. Those files have a well-defined structure and are automatically generated by the system, and so this provides all of the benefits of the Registry. However, because they are stored in individual files, they are not accessed until they are needed. If a preferences file is never needed (for example, if you have uninstalled the application) then it will never be loaded, and therefore it cannot possibly slow your system down.
If you really feel a need to remove old Preferences files, then as others have mentioned you can run a search for the application's name (or part of it). The system ensures that most preferences files contain the name of the app which created them, so this technique is actually quite effective. However, it is in no way harmful to leave these files on your system. They do not take significant amounts of space, and should you ever decide to reinstall the application, your old preferences will be preserved.