So I got a 2008 Xserve, did a clean install of 10.6 Server, and everything was fine. Then I shipped the server to my data center, and now it's becoming a huge disaster. In short, about 10% of the internet seems unable to resolve my domains and get to my server (I don't run my own DNS, some of my names are with Register and some are with GoDaddy). Also my server itself is unable to resolve most hosts. DNS service is enabled, so if I use my own IP for DNS, or if I use google's own public DNS, or I use any other 3rd party public DNS, I cannot resolve most hosts. Only a small amount.
The problem with incoming traffic may be solved just by waiting it out. But I think the outgoing traffic issue is more serious. It seems like the DNS lookup process itself is somehow broken. Even using google's DNS, I can't seem to resolve google.com. My old server running 10.5 Server is in the same data center. It has no issues with DNS. I even opened the firewall on my old server, and tried using it as my DNS server on the new server. But even that doesn't work. Nothing seems to work, or have any effect.
This is far from the first DNS problem I've been a part of with 10.6 Server. Getting an iCal server working without errors on 10.6 is a real nightmare, and AppleCare was really no help at all making iCal work. But my new server is old, so I couldn't call Apple about it anyway.
At this point, I am completely out of ideas to try. Note that the server is in a data center so I can't do anything that involves physical contact, like reinstall the OS. And also, I'm not trying to provide DNS services for any other machine. I just want to HAVE internet access on this server. And run the DNS service for any services on my server that need it (i.e. Mail).
This is interesting. I just resolved google.com and picked one of it's A records. I then pinged it on my home computer and got a steady 13ms ping time. I then pinged it from my server and got 100% time-outs. I just tried again with the firewall turned off, and got the same results.