Closed Bug 768283 Opened 13 years ago Closed 13 years ago

DNS records "wedged" on mv-ns*

Categories

(Infrastructure & Operations :: Infrastructure: Other, task)

Product:
Infrastructure & Operations
Component:
Infrastructure: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: bhourigan)

Details

10.250.0.{21,22} seem to have not noticed a zonefile update, but other nameservers have (mradm01, and its slaves). I poked around on the nameserver, and the zonefile is there and updated. The SOA has been bumped (twice). I ran 'rndc reload' on 10.250.0.21, but didn't restart named (which I'm sure would help, but shouldn't be necessary). [dustin@mvadm01 ~]$ dig -x 10.250.48.247 @10.250.0.21 ; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 @10.250.0.21 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45482 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3 ;; QUESTION SECTION: ;247.48.250.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 247.48.250.10.in-addr.arpa. 3600 IN PTR mv-production-puppet-old.build.mtv1.mozilla.com. ;; AUTHORITY SECTION: 48.250.10.in-addr.arpa. 3600 IN NS ns.ca.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.cn.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.mv.mozilla.com. ;; ADDITIONAL SECTION: ns.ca.mozilla.com. 3600 IN A 10.240.2.11 ns.cn.mozilla.com. 3600 IN A 10.241.2.11 ns.mv.mozilla.com. 3600 IN A 10.250.0.21 ;; Query time: 1 msec ;; SERVER: 10.250.0.21#53(10.250.0.21) ;; WHEN: Mon Jun 25 17:08:29 2012 ;; MSG SIZE rcvd: 213 [dustin@mvadm01 ~]$ dig -x 10.250.48.247 @10.250.0.22 ; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 @10.250.0.22 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10692 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3 ;; QUESTION SECTION: ;247.48.250.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 247.48.250.10.in-addr.arpa. 3600 IN PTR mv-production-puppet-old.build.mtv1.mozilla.com. ;; AUTHORITY SECTION: 48.250.10.in-addr.arpa. 3600 IN NS ns.ca.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.cn.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.mv.mozilla.com. ;; ADDITIONAL SECTION: ns.ca.mozilla.com. 3600 IN A 10.240.2.11 ns.cn.mozilla.com. 3600 IN A 10.241.2.11 ns.mv.mozilla.com. 3600 IN A 10.250.0.21 ;; Query time: 1 msec ;; SERVER: 10.250.0.22#53(10.250.0.22) ;; WHEN: Mon Jun 25 17:08:33 2012 ;; MSG SIZE rcvd: 213 [dustin@mvadm01 ~]$ dig -x 10.250.48.247 @10.250.48.19 ; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 @10.250.48.19 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5268 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0 ;; QUESTION SECTION: ;247.48.250.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 247.48.250.10.in-addr.arpa. 3600 IN PTR releng-puppet1.build.mtv1.mozilla.com. ;; AUTHORITY SECTION: 48.250.10.in-addr.arpa. 3600 IN NS ns.cn.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.mv.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.ca.mozilla.com. ;; Query time: 3 msec ;; SERVER: 10.250.48.19#53(10.250.48.19) ;; WHEN: Mon Jun 25 17:08:39 2012 ;; MSG SIZE rcvd: 155 [root@ns1a.build.mtv1 chroot]# dig -x 10.250.48.247 @10.2.74.123 ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6 <<>> -x 10.250.48.247 @10.2.74.123 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49438 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0 ;; QUESTION SECTION: ;247.48.250.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 247.48.250.10.in-addr.arpa. 3600 IN PTR releng-puppet1.build.mtv1.mozilla.com. ;; AUTHORITY SECTION: 48.250.10.in-addr.arpa. 3600 IN NS ns.ca.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.mv.mozilla.com. 48.250.10.in-addr.arpa. 3600 IN NS ns.cn.mozilla.com. ;; Query time: 7 msec ;; SERVER: 10.2.74.123#53(10.2.74.123) ;; WHEN: Mon Jun 25 17:10:48 2012 ;; MSG SIZE rcvd: 155
The SOA isn't updated, either: [dustin@mvadm01 ~]$ dig -x 10.250.48.247 -tSOA ; <<>> DiG 9.7.0-RedHat-9.7.0-1.rhel5 <<>> -x 10.250.48.247 -tSOA ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41653 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;247.48.250.10.in-addr.arpa. IN SOA ;; AUTHORITY SECTION: 48.250.10.in-addr.arpa. 1800 IN SOA ns.mv.mozilla.com. sysadmins.mozilla.org. 2012051500 10800 3600 604800 1800
Assignee: server-ops → server-ops-infra
Component: Server Operations → Server Operations: Infrastructure
QA Contact: phong → jdow
Assignee: server-ops-infra → bhourigan
:dustin It appears to be a bug in bind, 'rndc reload' is not picking up the changes after they've been updated via svn. I updated bind to the latest available on 5.5 and tried some other basic tricks but after an hour I was unable to get 'rndc reload' behaving as expected. Changing the 'service named reload' to 'service named restart' has fixed the problem. Bug 694195 exists to nuke and pave mv-ns01 and mv-ns02, so I expect that it will be functioning as intended on the new systems.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Component: Server Operations: Infrastructure → Infrastructure: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.