This is a long shot since no one uses this api anymore.
How does NavGetFile handle things if you use it to specify a filename containing foreign language unicode characters?
For example, I have a file containing the unicode character 0xAC74 (a korean character pronounced "gun"). The string that comes back is an infinity symbol and the << character. These map to B0 C7 in MacRoman.
If I select another unicode character such as 0x0100 (capital A with the long vowel symbol), the character actually comes back as some error number, #2A572.
You can see these characters as they're listed in the dialog, and even in OS 9's finder.