It turns out that our forwarder configuration is not working correctly, but due to split-horizon DNS this only affects nameserver lookups in PHX1 and not SCL2: ns1.phx1.svc$ dig @ns1.mozilla.org. metrics-logger1.private.scl3.mozilla.com. ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 29966 ns1.scl2.svc$ dig @ns1.mozilla.org. metrics-logger1.private.scl3.mozilla.com. ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44015 metrics-logger1.private.scl3.mozilla.com. 300 IN A 10.22.75.50 Use tcpdump to verify the forwarder issue and its eventual fix in staging and then deploy to production.
This may also be related to the PHX1 forwarder targets (10.8.75.21, 22) refusing lookups for the above hostname in PHX1. Will continue diagnosing once that's fixed (see "depends on" bug list above).
This turned out to be BIND9 converting "REFUSED" from the upstreams in PHX1 to "NXDOMAIN". SCL2 not affected. Revising description and resolving as fixed.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Summary: dns: forwarder configuration broken in phx1/scl2, affects only phx1 → dns: forwarder configuration broken in phx1
You need to log in before you can comment on or make changes to this bug.