762346 - DNS misconfiguration for phx1.mozilla.com.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Labs, task)

Description

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

There is an issue with the DNS configuration for phx1.mozilla.com. When I resolve pancake-elasticsearch1.labs.phx1.mozilla.com. I get the following response back:


[root@pancake-web4 supervisor]# dig pancake-elasticsearch1.labs.phx1.mozilla.com
;; QUESTION SECTION:
;pancake-elasticsearch1.labs.phx1.mozilla.com. IN A
;; ANSWER SECTION:
pancake-elasticsearch1.labs.phx1.mozilla.com. 300 IN A 10.8.110.21
;; AUTHORITY SECTION:
phx1.mozilla.com.	300	IN	NS	ns1.mozilla.org.
phx1.mozilla.com.	300	IN	NS	ns3.mozilla.org.
phx1.mozilla.com.	300	IN	NS	ns2.mozilla.org.
;; ADDITIONAL SECTION:
ns1.mozilla.org.	300	IN	A	63.245.215.5
ns2.mozilla.org.	300	IN	A	63.245.218.7
ns3.mozilla.org.	300	IN	A	63.245.212.5


So it gives back the right A record for pancake-elasticsearch1 but it also tells me that phx1.mozilla.com has three additional name servers: ns1 to ns3.mozilla.org.

However when I query those nameservers for pancake-elasticsearch1.labs.phx1.mozilla.com they give me an NXDOMAIN or a timeout in case of ns2:

[root@pancake-web4 supervisor]# host pancake-elasticsearch1.labs.phx1.mozilla.com. ns1.mozilla.org.
Host pancake-elasticsearch1.labs.phx1.mozilla.com. not found: 3(NXDOMAIN)

[root@pancake-web4 supervisor]# host pancake-elasticsearch1.labs.phx1.mozilla.com. ns2.mozilla.org.
;; connection timed out; no servers could be reached

[root@pancake-web4 supervisor]# host pancake-elasticsearch1.labs.phx1.mozilla.com. ns3.mozilla.org.
Host pancake-elasticsearch1.labs.phx1.mozilla.com. not found: 3(NXDOMAIN)

Comment 1

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

I think labs should be delegated?

[root@pancake-web4 supervisor]# host -t ns labs.phx1.mozilla.com ns1.mozilla.org
Host labs.phx1.mozilla.com not found: 3(NXDOMAIN)

Comment 2

[Philippe M. Chiasson (:gozer)]

Nope, DNS services are à global service.

Puntig this over to server ops

Comment 3

[Ashish Vijayaram [:ashish]]

Need more info. What NS is pancake-web4/others in Labs using? ns[1-3].mozilla.org should ideally point to internal IPs (pm-ns0[1,2]) within the Mozilla Infra.

Comment 4

[Philippe M. Chiasson (:gozer)]

[root@pancake-web4 ~]# cat /etc/resolv.conf 
search	labs.phx1.mozilla.com
nameserver	10.8.110.5

It seems to only be using a single NS 10.8.110.5 and I can't quite figure out what that is.

Comment 5

[Dumitru Gherman [:dumitru]]

Well I see that the machine has the DNS manually configured:

[root@pancake-elasticsearch1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 | grep DNS1
DNS1=10.8.110.5

If you keep those static, for PHX the nameservers are 10.8.75.21 and 10.8.75.22.
If you use DHCP, make sure you push those in the config.

Comment 7

[Philippe M. Chiasson (:gozer)]

(In reply to Dumitru Gherman [:dumitru] from comment #5)
> Well I see that the machine has the DNS manually configured:
> 
> [root@pancake-elasticsearch1 ~]# cat
> /etc/sysconfig/network-scripts/ifcfg-eth1 | grep DNS1
> DNS1=10.8.110.5
> 
> If you keep those static, for PHX the nameservers are 10.8.75.21 and
> 10.8.75.22.
> If you use DHCP, make sure you push those in the config.

Interesting, I didn't intially configure these hosts, so I'll need to fix this. No clue why they were setup with non-dhcp nameservers.

Comment 8

[Philippe M. Chiasson (:gozer)]

Done.