I'm confused by your post. XDarwin is just the implementation of XFree86 for Darwin/OSX. The only directories it will create during installation are /etc/X11 and /usr/X11R6 ("/dev" is the device directory). There are no "ports" of apps specific to XDarwin - X apps for OS X/Darwin should work fine for Apple's X11 or XDarwin (i.e. all of your current FINK installed X apps should work fine for either version of the X window system )
You can simply install XDarwin via fink (i.e. build from source) with "fink install xfree86-rootless", or you can go to sourceforge.net/projects/xonx and get the binary installs (double click on them, authenticate and continue as per any other OS X binary install).
There are specific instructions at the FINK web site about Apple's X11 versus XFree86/XDarwin (you'd be best to fully remove X11 before installing XDarwin, which would include purging your FINK installation of the placeholder "system-xfree86"). There are supposedly ways to actually run both on the same system, but I have not tried that.
As for panther, while I know Apple's X11 will be bundled with it, I believe it is an optional install. If you want it, you could just remove XDarwin - which can be very simple - just delete the app from "Applications", delete /etc/X11 and /usr/X11R6 - and then install Panther X11.