ATA/IDE hard drives have jumpers (usually located near the power connector) that allow you to control the mode of the drive. Depending on the drive model, the jumper configuration will be different. Most of the time, removing all the jumpers puts the drives in slave mode.
The primary hard drive already in your G4 is already configured as a master on the primary IDE bus. The optical drive is, if I recall, also configured as a master on the secondary IDE bus. This would mean that you would need to have your second hard drive be in slave mode, and attach it to one of the free IDE cable connectors.