Apple's version of X11 runs a window manager called quartz-wm. Windowmaker is also a window manager. Assuming you can get Apple's version of X11 to run a window manager other than quartz-wm, Windowmaker would have no problem managing your KDE apps.
Please note that running Windowmaker will only give your X11 apps NeXT-like window decorations, not widgets or menus. You'll need NeXT KDE and GTK themes and perhaps a replacement xaw library called neXtaw to get the widgets. The only way I know to get the NeXT vertical menus would be to use GNUstep.