649641 - time on some linux slaves is incorrect

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

dustin@lorentz ~ $ ssh cltbld@linux64-ix-slave05 date && date
Wed Apr 13 01:00:38 PDT 2011
Wed Apr 13 10:06:38 CDT 2011

CDT and PDT are not 9 hours apart :)

Does this call for NTP?  Or for setting the hardware clock?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I haven't checked any of the other linux64 builders - I just happened to notice on this one.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

linux-ix-slaveNN too:

[cltbld@linux-ix-slave08 slave]$ ssh -oBatchMode=no production-master01 date && date
Wed Apr 13 08:29:14 PDT 2011
Wed Apr 13 09:26:34 PDT 2011

Should we be running NTP on the slaves?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

needs triage..

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to comment #2)
> Should we be running NTP on the slaves?

Yes.  Cf bug 539278.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Bugs which break clobber are more important than P4, it turns out.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

*clobberer

So the issue is that we should disable ntp on linux VMs, since the VMWare tools will fight with you otherwise, but we need to enable it on hardware (eg ix). Could someone who knows about puppet suggest how
 http://hg.mozilla.org/build/puppet-manifests/file/default/modules/ntp/manifests/init.pp#l15
might be adapted (or just grab the bug).

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

We can probably use the 'virtual' facter fact in a case statement.

[cltbld@linux-ix-slave03 ~]$ facter virtual
physical

If I have a chance today, I'll grab this bug.

Comment 11

[Phil Ringnalda (:philor)]

And indeed http://tbpl.allizom.org/php/getParsedLog.php?id=6037003&full=1 is linux-ix-slave42 failing to pick up an 11:40 clobber because "our last clobber date ... 15:28:28"

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to John Ford [:jhford] from comment #10)
> We can probably use the 'virtual' facter fact in a case statement.
> 
> [cltbld@linux-ix-slave03 ~]$ facter virtual
> physical

Alas, no.

[cltbld@moz2-linux-slave30 ~]$ facter virtual
physical

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Works for me, now:

[root@moz2-linux-slave26 ~]# facter virtual
vmware

I just landed some patches to do this again in PuppetAgain: 
  http://hg.mozilla.org/build/puppet/rev/294b5dae3921
  http://hg.mozilla.org/build/puppet/rev/7410d33d3ec0

if you want to try to abstract any of that into the existing puppet-manifests

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet-manifests]

Turns out that facter is wrong when not run as root. This is an untested patch to do the right thing on VMs and real hardware.

Comment 15

[Justin Wood (:Callek)]

Comment on attachment 579909 [details] [diff] [review]
[puppet-manifests]

Nit we should either do an if/else for vmware or at the least do a default fail() or use the enable on default()

(incase facter ever returns anything else than vmware/physical here)

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

[cltbld@buildbot-master19 ~]$ facter virtual
kvm

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Disable ntp on vmware VMs, otherwise enable it

Tested with staging-puppet, it leaves VMs alone and turns on ntpd for ix boxes (and saw the clock get slewed to correctness on a couple of machines). Uses 'default' instead of 'physical' so we're covered if we ever move to (eg) kvm-based builders.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 590657 [details] [diff] [review]
Disable ntp on vmware VMs, otherwise enable it

http://hg.mozilla.org/build/puppet-manifests/rev/dbbf9ffae24a

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Deployed to the 3 masters for slaves. Clocks will slew to correct times in the few mins after ntpd starts, so if a slave picks up a job immediately there might be some funkyness around doing a clobber or not, but otherwise only make the slave's twistd.log funky.