Well, a lot (all?) of X11 is stored in /usr/X11R6, so you could just delete that whole directory. As for the stuff that's installed elsewhere, I don't know of a very clean way of deleting it. I suppose one way of really making sure would be doing an lsbom on the Archive.bom file inside the X11 installer package receipt in /Library/Receipts. Then you could run a quick script to delete all the files listed in that archive.
In any case, hosing /usr/X11R6 will certainly destroy a lot of it.