Actually, it looks like Apple have changed this since I last checked it (which was a while ago, back in 10.2).
Search for "DTS" in the help for DVD Player and you'll see the following information:
1. Connect your audio device to your computer.
2. If you are playing a DVD movie that provides DTS output and your audio device supports DTS, you can select it in the DVD disc menu.
3. Choose DVD Player > Preferences, and then click Disc Setup.
4. Choose the Digital Out item from the "Audio output" pop-up menu that includes the name of the digital audio device you want to use.
5. Click Play to start playing your movie.
- proton