1607289 - monitor time sync on mojave

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Posix OS, task)

Description

[:dhouse]

We cannot use ntpq on macos mojave to monitor time sync (was removed by apple).

Comment 1

[:dhouse]

• search results on stackoverflow/other show using snmp to do an active time server check
• man pages show apple's(not built-on/etc old timed) timed has no utilities or interface
• apple's timed keeps a plist file of "state" that has fields:

$ man timed | grep -A1 plist
     /var/db/timed/com.apple.timed.plist
                   The cached state of timed
--
     /System/Library/LaunchDaemons/com.apple.timed.plist
                   The timed service's property list file for launchd(8).
$ sudo defaults read /var/db/timed/com.apple.timed.plist
{
    TMCommandHistory =     (
    );
    TMLastNtpFetchAttempt = 0;
    TMLastRtcTime = "16.113918608";
    TMSystemTimeSet = 0;
    TMTimeSynthesizer = <62706c69 73743030 d4010203 [...] 01ba0000 00000000 02010000 00000000 00370000 00000000 00000000 00000000 01bc>;
    TMVersionKey = 15;
    UUID = <b9ad703a [...] 48c32674>;
}

I modified the time on a worker and watched for it to be re-set/corrected and it was not changed until reboot. The timed process was active, but may have a low priority and low check frequency.

Comment 2

[:dhouse]

re-trying to watch timed activity on fixing time. The timed log entries may give offset and adjustment details.

[[e-mail] ~]$ date; sudo date "1944.00"
Mon Jan  6 19:44:12 GMT 2020
Mon Jan  6 19:44:00 GMT 2020
[[e-mail] ~]$ sudo ls -la /var/db/timed/com.apple.timed.plist; sudo defaults read /var/db/timed/com.apple.timed.plist
-rw-r--r--  1 _timed  _timed  829 Jan  6 19:43 /var/db/timed/com.apple.timed.plist
{
    TMCommandHistory =     (
    );
    TMLastNtpFetchAttempt = 0;
    TMLastRtcTime = "16.857809807";
    TMSystemTimeSet = 0;
    TMTimeSynthesizer = <62706c69 73743030 [...]370000 00000000 00000000 00000000 01bc>;
    TMVersionKey = 15;
    UUID = <b9ad703a [...] 48c32674>;
}
[[e-mail] ~]$ log stream --predicate '(process MATCHES "timed")'
Filtering the log data using "process MATCHES "timed""
Timestamp                       Thread     Type        Activity             PID    TTL
2020-01-06 19:44:21.471231+0000 0x303      Default     0x1a3                76     0    timed: [com.apple.timed:data] cmd,fetchTime,num,3,result,3,mach,63353933229,tv_sec,140645732696192,tv_usec,1,delay,0.000000,dispersion,0.000000,more,0,ip,216.126.233.109,port,123,attempts,5,slept,0,use_service_port,1
2020-01-06 19:44:21.471315+0000 0x303      Default     0x1a3                76     0    timed: [com.apple.timed:text] NTP failed 4 times, switching off of low source port.
2020-01-06 19:44:21.473147+0000 0x1493     Default     0x1a3                76     0    timed: [com.apple.timed:data] cmd,SYNC,rtc_s,63.356322606,unc_s,0.000000000,mach,63356322606
2020-01-06 19:44:21.473239+0000 0x2a9      Default     0x1a3                76     0    timed: [com.apple.timed:text] Done bursting
2020-01-06 19:44:21.473313+0000 0x2a9      Default     0x1a3                76     0    timed: [com.apple.timed:data] cmd,checkActiveTimeSourceRequired,rtc_s,63.356543,bb_want_unc_s,inf,want_unc_s,0.050000,bb_want_rtc_s,-inf,want_rtc_s,-inf
2020-01-06 19:44:21.473347+0000 0x2a9      Default     0x1a3                76     0    timed: [com.apple.timed:text] Want active time in -0.00min. Remaining retry interval: 4.332686min.
2020-01-06 19:44:21.473372+0000 0x2a9      Default     0x1a3                76     0    timed: [com.apple.timed:text] Unscheduling wanted job
2020-01-06 19:44:21.473474+0000 0x2a9      Default     0x1a3                76     0    timed: [com.apple.timed:text] Scheduling wanted job

Comment 3

[:dhouse]

The timed logs show failures syncing with the ntp server we have configured (/etc/ntp.conf, "server 0.pool.ntp.org").
The macos_ntp ronin-puppet module defaults to time.apple.com, but we have a common data element that replaces it:
.//data/common.yaml:ntp_server: "0.pool.ntp.org"

Comment 4

[:dhouse]

it looks like we have an out-bound blocking on the ntp hosts (confirmed not localhost firewall).

[[e-mail] ~]$ sudo sntp -sS time.apple.com
sntp 4.2.8p10@1.3728-o Tue Mar 21 14:36:42 UTC 2017 (136.200.1~4588)
2020-01-06 21:13:28.768030 (+0000) +269.575792 +/- 179.717850 time.apple.com 17.253.26.125 s1 no-leap
[[e-mail] ~]$ sudo sntp -sS 0.pool.ntp.org
sntp 4.2.8p10@1.3728-o Tue Mar 21 14:36:42 UTC 2017 (136.200.1~4588)
0.pool.ntp.org 195.113.20.2 no UCST response after 5 seconds

Comment 5

[Jake Watkins [:dividehex]]

IIRC, all internal releng hosts should be set to ntp.build.mozilla.org. This CNAME will always point to a solid internal NTP clock, in this case being the hardware infoblox hosts in MDC.

Comment 6

[Jake Watkins [:dividehex]]

Attachment: GitHub Pull Request - use infoblox as time source

Comment 7

[:dhouse]

(In reply to Jake Watkins [:dividehex] from comment #5)

IIRC, all internal releng hosts should be set to ntp.build.mozilla.org. This CNAME will always point to a solid internal NTP clock, in this case being the hardware infoblox hosts in MDC.

When setting up the telegraf monitoring for mac-signing to match what was configured for nagios, I found that telegraf's time sync check required ntpd and so it would not work on mojave. I created this bug to add monitoring of the macos time sync (maybe to telegraf), and then that branched into fixing the time-sync since it was not working:

In ronin-puppet for macos and build-puppet for linux, the time sync was set to use ntp.org, and that has been firewall blocked. I don't think it was opened when first configured. I'm working through bug 1607546 to get that working. We can switch to using ntp.build.mozilla.org however. Is there a reason to use ntp.build.mozilla.org instead of ntp.org? Offhand, I'd gamble on ntp.org being available and correct instead of an internal host.

I discussed the time sync with Aki and he asked that the signing machines be sync'd to time.apple.com.
Is there a reason to not sync with apple's time across all of the mini's?

Comment 8

[Jake Watkins [:dividehex]]

(In reply to Dave House [:dhouse] from comment #7)

(In reply to Jake Watkins [:dividehex] from comment #5)

IIRC, all internal releng hosts should be set to ntp.build.mozilla.org. This CNAME will always point to a solid internal NTP clock, in this case being the hardware infoblox hosts in MDC.

When setting up the telegraf monitoring for mac-signing to match what was configured for nagios, I found that telegraf's time sync check required ntpd and so it would not work on mojave. I created this bug to add monitoring of the macos time sync (maybe to telegraf), and then that branched into fixing the time-sync since it was not working:

In ronin-puppet for macos and build-puppet for linux, the time sync was set to use ntp.org, and that has been firewall blocked. I don't think it was opened when first configured. I'm working through bug 1607546 to get that working. We can switch to using ntp.build.mozilla.org however. Is there a reason to use ntp.build.mozilla.org instead of ntp.org? Offhand, I'd gamble on ntp.org being available and correct instead of an internal host.

I discussed the time sync with Aki and he asked that the signing machines be sync'd to time.apple.com.
Is there a reason to not sync with apple's time across all of the mini's?

I think the notarization/signing services might be an exception here. But historically, we have always synced hosts to a local network clock. At one point, IT ran a couple stratum 1 NTP hosts and now the infoblox hardware I believe is running at stratum 2, which is fine. It is a common pattern to use local network NTP sources for networks with many hosts to sync to locally. This gives a more consistent time consensus among our hosts and workers than if they are all being pointed to many different third party reference clocks across the internet.

It probably wouldn't be a huge problem to sync all of our hosts to time.apple.com but I'd prefer to exercise more control over where our clocks reference their time from upstream. We control that with the infoblox hosts but not with time.apple.com.

Comment 9

[:dhouse]

np, I'm okay with using an internal ntp host. and I r+'d the PR.

Comment 10

[Jake Watkins [:dividehex]]

:aki, is there a strong argument to having the signing/notarization host time sync with apple's time servers instead of the local datacenter ntp servers?

Comment 11

[Aki Sasaki (not active)]

Assuming it works, local datacenter ntp syncing is good, possibly even preferable.

Comment 12

[Jake Watkins [:dividehex]]

I've set the default ntp source to ntp.build.mozilla.org which points at the infoblox box in MDC1. In the future, I'd like to add logic to actually set the ntp source based on location with an override for more special situations but for right now this works.

Comment 13

[Andrew Erickson [:aerickson]]

Clearing dhouse's NI's. Please NI me if this still needs attention.