455965 - time sync not working on aus2-staging

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We had trouble today when Tb2.0.0.17 updates were pushed to the beta channel, because the clock was fast on aus2-staging.m.o and it put a timestamp 25 mins into the future on the top-level dir of the update store. Over in bug 430737 we added a touch so that each AUS machine would invalidate its nfs cache when we added/updated the info. 

So, this is a request to 
* review the time syncing setup on aus2-staging.m.o, which is a VM. I think this box had an OS update fairly recently; not sure if the kernel argument clock=pit is still the way to go, but we're not using it now

* add a nagios check to make sure the time on this box stays accurate

Comment 1

[Dave Miller [:justdave]]

This machine is indeed a VM.  It does not have vmware-tools installed.  Also confirmed it does not have the kernel params in place.

It's running RHEL 5.2, which means the correct kernel param is actually "clocksource=pit". ("clock=pit" was a RHEL4 thing).

The host config *does* have syncTime enabled already, so in theory all we need to do is add clocksource=pit to the kernel params and reboot.

Is it safe to reboot this machine whenever?  Do we need to schedule an outage for it?

Comment 2

[Dave Miller [:justdave]]

grub.conf has been updated, all we need is an okay to reboot.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

You could reboot it now-ish. There's no release activity, and nightly builds are clustered in the wee small hours of PDT.

Comment 4

[Dave Miller [:justdave]]

Done.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Time is about 25 minutes slow on this box, looks like vmware-tools is not running:

$ sudo /etc/init.d/vmware-tools status
vmware-guestd is not running
$ sudo /etc/init.d/vmware-tools start
VMware Tools is installed, but it has not been 
(correctly) configured for the running kernel.
To (re-)configure it, invoke the following command: 
/usr/bin/vmware-config-tools.pl.

which needs to be run on the root console.

Comment 6

[Dave Miller [:justdave]]

the stuff in rc.local to rebuild the vmware drivers when the kernel gets updated was missing.  I just added it in and rebooted, looks like it's all set.