I don't see why a Radeon 9800XT wouldn't work in the rev. A machines... Don't know if anyone's tried it yet, but I can't imagine that it wouldn't work.
BTW, PCI and PCI-X have nothing to do with the AGP slot in to which the graphics boards are placed.
AGP (Advanced Graphics Port) is an Intel (IIRC) invention, which in a nutshell is basically a high capacity PCI interface running at up to 8x the speed of conventional PCI slots. There are also other optimizations designed for video cards. The connector is physically different than standard PCI slots, and the two aren't generally interchangeable. The G5s have one AGP slot. There are no machines, PC or Mac, that have more than one AGP slot.
PCI is the "current" standard for interface cards. PCI slots/cards run a parallel bus at either 33MHz or 66MHz. PCI-X is a higher speed variation of the PCI standard (100MHz and 133MHz). Most current, but not all, PCI cards will be compatible with PCI-X slots. The upper two G5 models have 3 slots that will accept PCI-X cards (two at 100MHz, one at 133MHz) or most conventional PCI cards (video cards, hard drive controllers, FireWire cards, USB cards, music/sound cards, etc...)
PCI Express (PCIe) is the successor to both PCI, PCI-X and AGP. Its a serial interface running at various multiples of the PCI standard (8x, 16x, etc). PCIe will be available in up coming high-end x86-based machines, and will probably eventually make its way in to a future version of the PowerMac. There are several other architectural benefits to PCIe... but I'll spare you the details (especially since I'm no electrical engineer).