Eh... that's a harder question than you might think. AVI and VOB are just container formats (as are Quicktime .mov and Microsofts .vmw and .asf) - it does not say anything about the actual streams inside the containers. You say that you want to play it on a DVD-player, so the output must be MPEG 2 for the video (with certain limitations on bitrate) and .ac3 or .mp2 (which is not MPEG 2, but MPEG 1 Audio Layer 2) for the audio. The .avi could, as I said, include anything under the sun. My best guess is that it's some sort of DivX or Xvid for the audio and .mp3 or .ac3 for the audio.
Your first stop should probably be
ffmpegx. That app should be able to do what you ask, but be aware that a) it takes time to reencode, even on a fast G4 or G5, b) it's not easy, so use the manual, the forums at that site and trial'n'error to figure it out and c) it may not work on your player, even though it works in the Mac. That was the case for my old player. I have since bought a new DVD player that plays my reencoded DVDs, but then it plays DivXs as well, so it's a moot point.
If you think that that is too hard, you might consider
Toast Titanium, which does the same thing but is a little easier to understand. It doesn't make better DVDs though - my old player did not accept these either, although I experimented less with it since I don't own it and had borrow a firends computer to try.
I've also heard that the latest version of VLC can do something like this, although I've never tried it.
All of these players can read antyhing that Quicktime can read, and in the case of ffmpegX, then some. They are not magical however - you can't read every file under the sun. Particularly Microsoft's stuff usually doesn't work.
Is that a good enough answer to get you started?