Due to bug 741112, I've had to reload named by hand on ns1/ns2.private.releng.scl3. In debugging this issue, I've noticed that ns2 has the IP of ns1 first in /etc/resolv.conf. A nameserver should always be listed first in its own resolv.conf file (usually as 127.0.0.1, but if it's not listening on that interface, then whatever interface is appropriate) to avoid the situation where it must wait for a timeout when it is unable to reach other nameservers.
I find it useful to include an options line similar to `options timeout:2 attempts:2` that helps with this problem.
Right now resolv.conf is pushed via dhcp - and the name server order is configured identically across all hosts. What would the reasoning be to require 127.0.0.1 as the first resolver?
:arr We have implemented this feature for you. rtucker added a key "nic.0.dhcp_domain_name_servers.0" to inventory which will set per host dns servers. Both ns1/ns2 in your business unit will now see 127.0.0.1 as the first resolver. I bounced networking on ns2 to test and verified it's working properly, but I have not done this on ns1 for fear of causing a minor interruption of DNS service. Once the lease expires it will pick up the changes.