Customized, high performance X servers are generally closed source. The next time someone says that X is slow, tell them to play with a SGI IRIX box.
However, the optimizations usually are hardware specific, and no hardware vendor wants everyone to be able to access the specs for their latest and greatest device. That's why nVidia only releases binary drivers for XFree86.