the extra files that an application creates when you first run it are the application's preferences. they are really small. you do not need to delete them, you can if you want to. i tend not to, since if for some reason i ever need that application again, it's settings are remembered.
~/Library/Preferences is where all these preference (.plist) files are stored.
applications might also create some files in the ~/Library/Application Support folder. take
proteus for example (first app that came to mind). it will create a proteus folder in the Application Support and then inside the proteus folder it will have folders for message themese, contact list icons, etc.
these files too can be left alone.
at the end of the day, it's your computer, you decide if you want to delete them or not.
you mentioned a pc world and the way an application gets installed. like you said, an application installs and throws files all over the place. nearly all of the applications you will install on your mac will be one file (a container) with all it's files inside it. drag it to your Applications folder (or where ever you choose) and it is installed, drag it to the trash, and it is uninstalled.
control click > show package contents, on an application and look around inside it, it's kinda interesting.