bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

DNS misconfiguration for phx1.mozilla.com.

RESOLVED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Labs
RESOLVED FIXED
6 years ago
2 years ago

People

(Reporter: st3fan, Assigned: gozer)

Tracking

Details

(Whiteboard: )

(Reporter)

Description

6 years ago
There is an issue with the DNS configuration for phx1.mozilla.com. When I resolve pancake-elasticsearch1.labs.phx1.mozilla.com. I get the following response back:


[root@pancake-web4 supervisor]# dig pancake-elasticsearch1.labs.phx1.mozilla.com
;; QUESTION SECTION:
;pancake-elasticsearch1.labs.phx1.mozilla.com. IN A
;; ANSWER SECTION:
pancake-elasticsearch1.labs.phx1.mozilla.com. 300 IN A 10.8.110.21
;; AUTHORITY SECTION:
phx1.mozilla.com.	300	IN	NS	ns1.mozilla.org.
phx1.mozilla.com.	300	IN	NS	ns3.mozilla.org.
phx1.mozilla.com.	300	IN	NS	ns2.mozilla.org.
;; ADDITIONAL SECTION:
ns1.mozilla.org.	300	IN	A	63.245.215.5
ns2.mozilla.org.	300	IN	A	63.245.218.7
ns3.mozilla.org.	300	IN	A	63.245.212.5


So it gives back the right A record for pancake-elasticsearch1 but it also tells me that phx1.mozilla.com has three additional name servers: ns1 to ns3.mozilla.org.

However when I query those nameservers for pancake-elasticsearch1.labs.phx1.mozilla.com they give me an NXDOMAIN or a timeout in case of ns2:

[root@pancake-web4 supervisor]# host pancake-elasticsearch1.labs.phx1.mozilla.com. ns1.mozilla.org.
Host pancake-elasticsearch1.labs.phx1.mozilla.com. not found: 3(NXDOMAIN)

[root@pancake-web4 supervisor]# host pancake-elasticsearch1.labs.phx1.mozilla.com. ns2.mozilla.org.
;; connection timed out; no servers could be reached

[root@pancake-web4 supervisor]# host pancake-elasticsearch1.labs.phx1.mozilla.com. ns3.mozilla.org.
Host pancake-elasticsearch1.labs.phx1.mozilla.com. not found: 3(NXDOMAIN)
(Reporter)

Comment 1

6 years ago
I think labs should be delegated?

[root@pancake-web4 supervisor]# host -t ns labs.phx1.mozilla.com ns1.mozilla.org
Host labs.phx1.mozilla.com not found: 3(NXDOMAIN)
(Assignee)

Comment 2

6 years ago
Nope, DNS services are à global service.

Puntig this over to server ops
Assignee: gozer → server-ops
Component: General → Server Operations
Product: Pancake → mozilla.org
QA Contact: general → phong
Target Milestone: M3 → ---
Version: unspecified → other
Need more info. What NS is pancake-web4/others in Labs using? ns[1-3].mozilla.org should ideally point to internal IPs (pm-ns0[1,2]) within the Mozilla Infra.
(Assignee)

Comment 4

6 years ago
[root@pancake-web4 ~]# cat /etc/resolv.conf 
search	labs.phx1.mozilla.com
nameserver	10.8.110.5

It seems to only be using a single NS 10.8.110.5 and I can't quite figure out what that is.
Severity: major → normal
Well I see that the machine has the DNS manually configured:

[root@pancake-elasticsearch1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1 | grep DNS1
DNS1=10.8.110.5

If you keep those static, for PHX the nameservers are 10.8.75.21 and 10.8.75.22.
If you use DHCP, make sure you push those in the config.
Assignee: server-ops → server-ops-labs
Component: Server Operations → Server Operations: Labs
QA Contact: phong → zandr

Updated

6 years ago
Duplicate of this bug: 762342
(Assignee)

Comment 7

6 years ago
(In reply to Dumitru Gherman [:dumitru] from comment #5)
> Well I see that the machine has the DNS manually configured:
> 
> [root@pancake-elasticsearch1 ~]# cat
> /etc/sysconfig/network-scripts/ifcfg-eth1 | grep DNS1
> DNS1=10.8.110.5
> 
> If you keep those static, for PHX the nameservers are 10.8.75.21 and
> 10.8.75.22.
> If you use DHCP, make sure you push those in the config.

Interesting, I didn't intially configure these hosts, so I'll need to fix this. No clue why they were setup with non-dhcp nameservers.
Assignee: server-ops-labs → gozer
Status: NEW → ASSIGNED
(Assignee)

Comment 8

6 years ago
Done.
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Whiteboard:
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.