A G4, which by the way will either come with Mac OS 9, or upgradeable for <$20, has 3 hard drive bays, one of which is occupied by the original drive. There are a total of 4 IDE connectors, one each for the standard ATA/66 drive it comes with, the CD-ROM, DVD-ROM or DVD-RAM, and the optional internal zip drive. So you may have only one connector left. It should work, but honestly I would advise against it unless it's just to copy the files over real quick. A drive from the time of Windows 3.1 will be quite slow, and will slow the system down every time it is accessed (which Mac OS does regularly, even when not actually accessing user data on that drive).
I am not *certain* as to whether a FAT volume will mount, although I suspect it would. No software should be necessary. Again, I'd avoid using this as a long-term fix, but connecting it just to copy the files real quick should be fine.
In any case, you would need to know how to change the jumpers on the drive to set it to "slave" mode.
tooki