763684 - update-named issue: "service named reload" does not load new SOA/zone information

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[:Atoll]

Our nameservers stopped loading new zone data after an update-named change from "service named restart" to "service named reload" recently.

Per cronmail, everything worked just fine:

Fetching external item into 'zones'
U    zones/mozilla.com/svc/phx1
U    zones/mozilla.com/svc/SOA
Updated external to revision 40464.
...
Reloading named: [  OK  ]

But it simply didn't load the new SOA data until a "service named restart" was run manually on the server.

This affects all nameservers in SCL2 and PHX1.

Comment 1

[:Atoll]

Also ignores failure result code from named reload:

Reloading named:                                           [FAILED]
After reload:
total 56
...

Implementing puppet code to test sysadmins/dnsconfig-weave/bin/update-named.stage on ns1/2.scl2.stage.svc

Comment 2

[:Atoll]

Attachment: use rndc freeze+reload+thaw instead of rndc reload, so that dynamic zones are reloaded as well

Root cause is that some or all of our zones are considered dynamic, and so "rndc reload" only reloads non-dynamic zones. Attempting to "rndc reload soatest.svc.mozilla.com" logs:

rndc: 'reload' failed: dynamic zone

to ensure that we reload zones correctly, the following alteration to update-named.stage repairs the issue:

-          /sbin/service named reload
+          "$RNDC freeze; $RNDC reload; $RNDC thaw;"

This patch implements the above change, and is confirmed working in stage using "soatest.svc.mozilla.com" zone which exists only in stage configs.

Comment 3

[:Atoll]

deployed to SCL2, PHX1