763374 - ns2[a-b].pubdmz.phx network time not updating

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Rick Bryce [:rbryce]]

During the phx outage today, all servers Time Sync check recovered but these two.

ns2b.pubdmz.phx1.mozilla.com:Time Sync is CRITICAL: NTP CRITICAL: No response from NTP server

ns2a.pubdmz.phx1.mozilla.com:Time Sync is CRITICAL: NTP CRITICAL: No response from NTP server
--

Ntpd appears to be running, but not updating from .  I manually set the times to be close to the ntp server thinking they may be to far our for a sync.  Not sure what else todo.

Comment 1

[Rick Bryce [:rbryce]]

Ntpd appears to be running, but not updating from time server.

Comment 2

[Jake Maul [:jakem]]

Best I can figure, these servers can no longer reach 10.8.75.9 (ip-admin02.phx.mozilla.org) on 123/udp, which is the only configured NTP server in ntp.conf. Or perhaps it can make the query, but not receive the response.

This is relatively easy to test by stopping ntpd and running "ntpdate 10.8.75.9"...

[[e-mail]1 ~]# ntpdate 10.8.75.9
12 Jun 10:38:09 ntpdate[3783]: no server suitable for synchronization found

I don't know what's changed, but this server doesn't even have an internal IP... it's external-only. So it's no surprise it can't query an internal IP. Maybe some sort of puppet manifest breakage caused this to revert from hitting some non-Mozilla IPs to some default ntp.conf?

Comment 3

[Adrian J Fernandez [:Aj]]

Confirmed working now (netflow related, not sure how it was previously removed/changed);

ns2a.pubdmz.phx1.mozilla.com:Time Sync is OK - NTP OK: Offset -0.6096582413 secs
ns2b.pubdmz.phx1.mozilla.com:Time Sync is OK - NTP OK: Offset -0.01031970978 secs