That's not Safari's fault. The web server is probably configured wrongly, so that it tells Safari the file is a text file. Safari then adds the .txt extension because it knows that the "text file" would otherwise be opened with Quicktime (and it cannot know that would be right.)
With Apache, e.g., you have to edit the configuration file and add the command:
AddType video/mp4 .mp4
if the file has the .mp4 extension. If you cannot change the configuration of Apache, because it's not your own server, then you can still put the line in a new text file called ".htaccess" and place that text file in the same directory as your video file. (You should call the file "htaccess" first and rename it to ".htaccess" on the server, because Mac OS X doesn't let you rename files to something with a dot at the beginning.)
If your server doesn't run Apache, you should look in the documentation to see how to add mimetypes.