How are you timing the DNS lookup? Just anecdotal observations in a browser, or at the command line?
Try running nslookup in a terminal window, and see how long it takes there.
You might find that your primary DNS server has a problem and the system is waiting for that to time out before rolling over to the secondary system.
If it runs quickly using nslookup in a terminal window, then it's not a problem with DNS per se. rather what the Mac is doing before it uses DNS.
Check the Directory Access app in /Applications/Utilities and turn off any lookup methods you're not using (e.g. LDAP). The system might be checking with these directory services before falling back to DNS.
One other troubleshooting option that might help is running a tcpdump in a terminal window when you experience the problem. Although somewhat hard to decypher, it might indicate what the Mac is doing - for example, if you see the DNS lookup packet go out to the DNS server, but a long paused before you see the reply, that would indicate a problem on the DNS server side and not with your Mac.