1114606 - Switch ntp servers away from time.mozilla.org

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Pete Moore [:pmoore][:pete]]

We are currently using time.mozilla.org for ntp on our buildbot masters and various other machines, but we should be using http://www.pool.ntp.org/zone/us

Today when bug 1114255 was rolled out, many of our machines went bananas (see https://bugzilla.mozilla.org/show_bug.cgi?id=1114582#c1).

:digi has recommended we migrate. Probably we need to do this to unblock bug 1114255.

Comment 1

[Brian Hourigan [:digi]]

It's OK to use time.mozilla.org as one of the servers, but it should not be the only one. We currently don't have an SLA for this service and it's purpose (up until now) was to donate resources to the US NTP pool.

Comment 2

[Pete Moore [:pmoore][:pete]]

Thanks Brian.

It looks like this is where we define our ntp settings:

http://hg.mozilla.org/build/puppet/file/5a4fa0e0da97/manifests/moco-config.pp#l67
http://hg.mozilla.org/build/puppet/file/5a4fa0e0da97/manifests/relabs-config.pp#l60

Comment 3

[Justin Wood (:Callek)]

fwiw we were told (over 6 years ago) to avoid using the ntp pool, and to use time.m.o

I however, cannot recall what those reasons were.

In addition:
http://mxr.mozilla.org/build/source/puppet/setup/masterize/00-setup.sh#30 as well, and there is http://mxr.mozilla.org/build/source/puppet/manifests/qa-config.pp#39 which we might want to check if its still valid as well

Comment 4

[Brian Hourigan [:digi]]

(In reply to Justin Wood (:Callek) from comment #3)
> fwiw we were told (over 6 years ago) to avoid using the ntp pool, and to use
> time.m.o

Yeah I think there is a lot of ambiguity here, I do want to mention this change does not have to be permanent. We have an urgent need to unblock bug 1114255, and we do not have any other publicly accessible NTP service.

We would ask that you use the global pool for now (barring any security concerns) and we can revisit this after the holidays and come up with a more permanent solution.

> In addition:
> http://mxr.mozilla.org/build/source/puppet/setup/masterize/00-setup.sh#30 as
> well, and there is
> http://mxr.mozilla.org/build/source/puppet/manifests/qa-config.pp#39 which
> we might want to check if its still valid as well

That is indeed valid, although, you should be using both ns1.private.scl3 and ns2.private.scl3 as sources.

Comment 5

[Justin Wood (:Callek)]

Attachment: [puppet] use pool.ntp

So the puppet setting is *not* http:// (and per http://www.pool.ntp.org/zone/us ) it should be a list of #.us.pool.ntp.org, however ntp pools have a nice benefit of allowing me to specify a single us.pool.ntp.org to do this.


[[e-mail] ~]# /usr/sbin/ntpdate -v us.pool.ntp.org
22 Dec 09:10:38 ntpdate[22845]: ntpdate 4.2.4p8@1.1612-o Tue Nov 29 00:09:13 UTC 2011 (1)
22 Dec 09:10:39 ntpdate[22845]: adjust time server 74.207.242.71 offset 0.000861 sec

I've added the us pool to seamonkey (all our hosts are in scl3 atm) and touched relabs and relops, I did not touch QA since I'm told its still valid.

Followup bug to be filed on supporting a pool of hosts for ntp.

Comment 6

[Justin Wood (:Callek)]

https://hg.mozilla.org/build/puppet/rev/8c77b56b5660
https://hg.mozilla.org/build/puppet/rev/9e5661947b4f

Should take at most 1hour to propagate, usually within 30 min

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Two potential issues here.

First, we're only using a single upstream server from the pool, so we're in the same situation we have been -- that server can be turned off without warning.  For example:

ntpq> pee
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*clock.team-cymr 204.123.2.72     2 u  998 1024  377   63.066   -3.898   5.107

I don't recall if NTP clients will automatically re-resolve a DNS name when one fails (so in the example above, if clock.team-cymru.org fails, will ntpd re-resolve us.pool.ntp.org?).  :digi, do you know?


Second, the checks in nagios are still comparing to ntp.build.mozilla.org:
            check_command => 'check_ntp_time!ntp.build.mozilla.org!3!60',

[[e-mail] dmitchell]# host ntp.build.mozilla.org
ntp.build.mozilla.org is an alias for time.mozilla.org.
time.mozilla.org has address 63.245.214.136
time.mozilla.org has address 63.245.214.135

which means that once that service is again not available, we'll see the same check failures, even though the hosts themselves continue to sync properly against a different server.  The NTP check and the NTP client should be using the same upstream server(s).


Depending on the answer to the re-resolution problem, the fixes here are:

YES:
 * configure hosts to use ntp.build.mozilla.org
 * change ntp.build.mozilla.org to an A pointing to ns{1,2}.private.scl3

NO:
 * reconfigure the NTP puppet module to allow multiple servers to be specified
 * specify both ns{1,2}.private.scl3

:digi, does that make sense?  Is there some other host or hostname that is the current best practice NTP upstream?  Or is using the public pool really best (that seems sort of crazy to me)?

Comment 8

[Jake Watkins [:dividehex]]

(In reply to Dustin J. Mitchell [:dustin] from comment #7)

> 
> Depending on the answer to the re-resolution problem, the fixes here are:
> 
> YES:
>  * configure hosts to use ntp.build.mozilla.org
>  * change ntp.build.mozilla.org to an A pointing to ns{1,2}.private.scl3
> 
> NO:
>  * reconfigure the NTP puppet module to allow multiple servers to be
> specified
>  * specify both ns{1,2}.private.scl3

I don't remember if NTP re-resolves, but I do know that NTP was designed to be aware many ntp hosts and the ntp client should be the one to decide which host to peer with based on it own algorithms.  Also, I don't think every client should be relying on an external pool.  This would mean that us.pool.ntp.org could resolve to many different ntp sources for all our hosts.  Internal hosts should be configured with multiple internal ntp services such as ns{1,2}.private.scl3. So my vote would be for the latter.

Comment 9

[Brian Hourigan [:digi]]

> First, we're only using a single upstream server from the pool, so we're in
> the same situation we have been -- that server can be turned off without
> warning.  For example:

The immediate need here was to migrate away from tick/tock, as we had to shut them down due to security concerns.

> I don't recall if NTP clients will automatically re-resolve a DNS name when
> one fails (so in the example above, if clock.team-cymru.org fails, will ntpd
> re-resolve us.pool.ntp.org?).  :digi, do you know?

I'm not positive on the DNS resolution behavior, I poked at the source, documentation, and mailing lists and it's still not clear. The anecdotes I've seen so far suggest that ntpd will call gethostbyname() on the configured server and libresolv picks 1 address to return back to ntpd.

The right answer here depends on the use case. If ntpd is persistent and it has the luxury of waiting until it retries, then having 1 configured host seems appropriate. If the use case is that this is for buildbots when they boot and it needs an answer the first time every time then we need to configure multiple servers.

> :digi, does that make sense?  Is there some other host or hostname that is
> the current best practice NTP upstream?  Or is using the public pool really
> best (that seems sort of crazy to me)?

After operating two stratum 1 devices and contributing to the US NTP pool I personally have a high degree in confidence in their operations. There are numerous safety checks and it's pretty well maintained. The project isn't stagnant and Ask Bjørn is extremely responsive.

You're welcome to use ns[12].private.(scl3|phx1) as long as you only require 1 server to be up. FYI these servers get their time from the US NTP pool.

Another thing to keep in mind - if any NTP consumers are located outside of SCL3 you should point to both phx1 and scl3, otherwise the tunnel will be a SPOF.

Comment 10

[Jake Watkins [:dividehex]]

As discussed in the Relops meeting today, we are going to rewrite ntp module to handle more than one ntp peer in the configuration.  We will then point these at ns1 and ns2 A records.  Since pandas are a little more difficult to reconfigure, ntp.build cname will be pointed at ns1.  It isn't a very robust option but panda are already fragile.

Dustin will handle fixing the ntp monitoring in another bug.

Comment 11

[Jake Watkins [:dividehex]]

As confirmed by :markco, all windows hosts are using the domain controllers as their clock source. This is fine but I'd like to ensure the DCs are getting their time from the same source as the rest of releng.

Which will be:
ns1.private.releng.scl3.mozilla.com
ns2.private.releng.scl3.mozilla.com

:Q, is this configured via gpo and can we set this on the DCs?

Comment 12

[Jake Watkins [:dividehex]]

Attachment: bug1114606-1.patch

This patch shifts the ntp module from single ntp clock source server to having multiple servers which will provide better sampling diversity and redundancy. It also changes the servers used for some orgs but most significantly points releng to internal ntp servers (ns1.releng and ns2.releng) instead of the external ntp pool.

Tested on ubuntu and osx.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8552538 [details] [diff] [review]
bug1114606-1.patch

Review of attachment 8552538 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/config/manifests/base.pp
@@ +86,5 @@
>      $puppetmaster_upstream_rsync_source = ''
>      ##
>      ## basic host configuration
>      ##
>      # NTP server to use for time sync

pluralize the comment

Comment 14

[Jake Watkins [:dividehex]]

Comment on attachment 8552538 [details] [diff] [review]
bug1114606-1.patch

remote:   https://hg.mozilla.org/build/puppet/rev/0db459844ddd
remote:   https://hg.mozilla.org/build/puppet/rev/f63f5193e5d8

Comment 15

[Jake Watkins [:dividehex]]

I double checked the panda squashfs image and confirmed pandas are using NTPSERVERS="ntp.build.mozilla.org" during the mozpools pxe boot state (during re-imaging, selftest, etc)

ntp.build.mozilla.org currently points time.mozilla.org which has 2 A records resolving to tick and tocks public ip.  I'm going to change that to point to ns1.private.releng.scl3.mozilla.com

Comment 16

[Jake Watkins [:dividehex]]

ntp.build.m.o has been changed.  Pandas should be the only hosts using this cname.


;; ANSWER SECTION:
ntp.build.mozilla.org.  60      IN      CNAME   ns1.private.releng.scl3.mozilla.com.

[[e-mail] ~]# ntpq -p ntp.build.mozilla.org
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*x.ns.gin.ntt.ne 129.250.36.10    2 u  124 1024  377    2.179    0.898   0.448
-tick.nullmodem. 128.252.19.1     2 u  174 1024  377   57.355   -2.368   1.138
+host2.kingrst.c 129.6.15.29      2 u  728 1024  377   64.017    2.503   0.515
-mail.honeycomb. 152.2.21.1       3 u  359 1024  377   58.685   -1.216   0.491
 ns1.private.rel .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+ns2.private.rel 204.2.134.163    3 u  714 1024  376    0.542    1.170   1.288
 LOCAL(0)        .LOCL.          10 l  82h   64    0    0.000    0.000   0.000

Comment 17

[Jake Watkins [:dividehex]]

I've filed 1126870 to improve ntp peering across all of our infrastructure.  When this is implemented, we can roll out a larger list of internal ntp servers and let ntpd decide best clock source just as the ntp god intended it.

Comment 18

[Q]

(In reply to Jake Watkins [:dividehex] from comment #11)
> As confirmed by :markco, all windows hosts are using the domain controllers
> as their clock source. This is fine but I'd like to ensure the DCs are
> getting their time from the same source as the rest of releng.
> 
> Which will be:
> ns1.private.releng.scl3.mozilla.com
> ns2.private.releng.scl3.mozilla.com
> 
> :Q, is this configured via gpo and can we set this on the DCs?

Jake back to your previous comment the dc's should be syncing to those sources today.

Comment 19

[Jake Watkins [:dividehex]]

This majority of the scope of work here was completed back in January.  I've opened bug1146719 for the last outstanding issue.