Closed Bug 768283 Opened 12 years ago Closed 12 years ago

DNS records "wedged" on mv-ns*

Categories

(Infrastructure & Operations :: Infrastructure: Other, task)

Product:
Infrastructure & Operations
Component:
Infrastructure: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: bhourigan)

Details

10.250.0.{21,22} seem to have not noticed a zonefile update, but other nameservers have (mradm01, and its slaves).  I poked around on the nameserver, and the zonefile is there and updated.  The SOA has been bumped (twice).

I ran 'rndc reload' on 10.250.0.21, but didn't restart named (which I'm sure would help, but shouldn't be necessary).

[dustin@mvadm01 ~]$ dig -x 10.250.48.247 @10.250.0.21

; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 @10.250.0.21
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45482
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3

;; QUESTION SECTION:
;247.48.250.10.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
247.48.250.10.in-addr.arpa. 3600 IN     PTR     mv-production-puppet-old.build.mtv1.mozilla.com.

;; AUTHORITY SECTION:
48.250.10.in-addr.arpa. 3600    IN      NS      ns.ca.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.cn.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.mv.mozilla.com.

;; ADDITIONAL SECTION:
ns.ca.mozilla.com.      3600    IN      A       10.240.2.11
ns.cn.mozilla.com.      3600    IN      A       10.241.2.11
ns.mv.mozilla.com.      3600    IN      A       10.250.0.21

;; Query time: 1 msec
;; SERVER: 10.250.0.21#53(10.250.0.21)
;; WHEN: Mon Jun 25 17:08:29 2012
;; MSG SIZE  rcvd: 213

[dustin@mvadm01 ~]$ dig -x 10.250.48.247 @10.250.0.22

; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 @10.250.0.22
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10692
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3

;; QUESTION SECTION:
;247.48.250.10.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
247.48.250.10.in-addr.arpa. 3600 IN     PTR     mv-production-puppet-old.build.mtv1.mozilla.com.

;; AUTHORITY SECTION:
48.250.10.in-addr.arpa. 3600    IN      NS      ns.ca.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.cn.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.mv.mozilla.com.

;; ADDITIONAL SECTION:
ns.ca.mozilla.com.      3600    IN      A       10.240.2.11
ns.cn.mozilla.com.      3600    IN      A       10.241.2.11
ns.mv.mozilla.com.      3600    IN      A       10.250.0.21

;; Query time: 1 msec
;; SERVER: 10.250.0.22#53(10.250.0.22)
;; WHEN: Mon Jun 25 17:08:33 2012
;; MSG SIZE  rcvd: 213

[dustin@mvadm01 ~]$ dig -x 10.250.48.247 @10.250.48.19

; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 @10.250.48.19
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5268
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0

;; QUESTION SECTION:
;247.48.250.10.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
247.48.250.10.in-addr.arpa. 3600 IN     PTR     releng-puppet1.build.mtv1.mozilla.com.

;; AUTHORITY SECTION:
48.250.10.in-addr.arpa. 3600    IN      NS      ns.cn.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.mv.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.ca.mozilla.com.

;; Query time: 3 msec
;; SERVER: 10.250.48.19#53(10.250.48.19)
;; WHEN: Mon Jun 25 17:08:39 2012
;; MSG SIZE  rcvd: 155


[root@ns1a.build.mtv1 chroot]# dig -x 10.250.48.247 @10.2.74.123

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6 <<>> -x 10.250.48.247 @10.2.74.123
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49438
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0

;; QUESTION SECTION:
;247.48.250.10.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
247.48.250.10.in-addr.arpa. 3600 IN     PTR     releng-puppet1.build.mtv1.mozilla.com.

;; AUTHORITY SECTION:
48.250.10.in-addr.arpa. 3600    IN      NS      ns.ca.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.mv.mozilla.com.
48.250.10.in-addr.arpa. 3600    IN      NS      ns.cn.mozilla.com.

;; Query time: 7 msec
;; SERVER: 10.2.74.123#53(10.2.74.123)
;; WHEN: Mon Jun 25 17:10:48 2012
;; MSG SIZE  rcvd: 155
The SOA isn't updated, either:

[dustin@mvadm01 ~]$ dig -x 10.250.48.247 -tSOA 

; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 -tSOA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41653
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;247.48.250.10.in-addr.arpa.    IN      SOA

;; AUTHORITY SECTION:
48.250.10.in-addr.arpa. 1800    IN      SOA     ns.mv.mozilla.com. sysadmins.mozilla.org. 2012051500 10800 3600 604800 1800
Assignee: server-ops → server-ops-infra
Component: Server Operations → Server Operations: Infrastructure
QA Contact: phong → jdow
Assignee: server-ops-infra → bhourigan
:dustin

It appears to be a bug in bind, 'rndc reload' is not picking up the changes after they've been updated via svn. I updated bind to the latest available on 5.5 and tried some other basic tricks but after an hour I was unable to get 'rndc reload' behaving as expected.

Changing the 'service named reload' to 'service named restart' has fixed the problem. Bug 694195 exists to nuke and pave mv-ns01 and mv-ns02, so I expect that it will be functioning as intended on the new systems.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Server Operations: Infrastructure → Infrastructure: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.