I would not upgrade it if I where you. Cost add up fast, and it simply does not pay. The worst thing about upgrading 604 boards is the video. They always have problems dealing with the increased processor speeds. QE also does not support PCI and apple is not keen on supporting these machines much longer. I would simply leave it as is, its not all that slow really even running X.
If you want to build yourself a cheap mac, I sugest the oldest one to look into is a sawtooth G4. (Even better a Gigabit, because it supports dual processors)
You can upgrade everyting really cheap and its ALMOST modern...
At 400 mhz its still acceptable and processor upgrades are semi cheap and droping...
Flashed PC radeon 8500 - $86.00
Superdrive (DVR-104) - $250.00
LARGE IDE hard drives $100+
800 mhz processor - $450
And you have a new old school machine that can keep up with the new kids.. Kinda. At least you can do everything in X you can with a new mac.
I have a Sawtooth with everything listed above, except the Processor and Id have to say I am pretty happy with it, and have no desire for a new computer just yet.
In short, I find it a very bad Idea to upgrade 604 boards. Memory upgrade is all I would invest in.