360005 - Time on various VMs is incorrect

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[J. Paul Reed [:preed]]

The time, as reported by date, on various VMs is incorrect.

This was the case with (most importantly right now) tinderbox, but also bm-admin01 is off by hours.

Since these are VMs, running NTP isn't optimal. NTP needs to be turned on and synced on the *host* machine, and then time sync in the VMware Tools needs to be turned on, and the VM needs to be rebooted for these settings to take effect.

3.0 may be different, but I've got this working beautifully with the VMs on ESX 2.5.x in the build farm. Let me know if you'd like some help setting it up.

This is a critical bug because, for instance, it's important that tinderbox/bonsai display the correct time.

Comment 1

[Dave Miller [:justdave]]

And as mentioned on IRC, I can't help with this because the VMWare admin stuff on 3.0 requires Windows, and I don't have access to a Windows machine.

Comment 2

[Justin Fitzhugh]

mrz and preed worked on this today.  Preed thinks the issue is that the vm wasn't sync'd first, and now that it is, it should hold time.  We are watching bm-admin01 until tomorrow to see what happens.

Comment 3

[J. Paul Reed [:preed]]

Checked today; time has jumped forward.

I think we're hitting the problem described in: http://kb.vmware.com/vmtnkb/search.do?cmd=displayKC&docType=kc&externalId=1420&sliceId=SAL_Public

where time is moving faster, not slower, so the tools aren't correcting for it as they should.

They give some kernel options to pass to the 2.6 kernel to try to help; we should attempt to play with those. We can use bm-admin01 for testing (as long as it's not down for more than a few minutes between reboots) if you'd like.

Comment 4

[Justin Fitzhugh]

I opened a ticket with VMWare on the right way to do this in ESX 3.0.  Will report back.

Comment 5

[Justin Fitzhugh]

This will be fixed in 2.6.18, until then they say to run a cron job every hour to sync time :(  Please add this to your VMs

-----

The Linux kernel change that Bobbi had talked about is in 2.6.18 kernel

http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.18

Do a search for "[PATCH] time: fix time going backward w/ clock=pit"

This was the patch submitted to correct the time drift issue with RHEL VM's.
I know the above line states the time going backwards but your problem with
the time going forward could be due to no clock= option being set in the VM.

Looking at the kernel sources provided by RedHat from their ftp server I
can't tell that this patch has been added to their kernels.

Until RedHat adds this patch to the kernel they ship the time will not sync
correctly and you will have to setup a cron job to sync the time with ntp
every 5 to 10 minutes depending on the drift you see.

I will be out of the office for the rest of the week. We can speak on Monday
if you like.

Comment 6

[J. Paul Reed [:preed]]

(In reply to comment #5)
> This will be fixed in 2.6.18, until then they say to run a cron job every hour
> to sync time :(  Please add this to your VMs

Hrm... below, they say 

> This was the patch submitted to correct the time drift issue with RHEL VM's.
> I know the above line states the time going backwards but your problem with
> the time going forward could be due to no clock= option being set in the VM.

Did you try setting a clock= option to test if that fixes the problem?

The problem was time going forward, not backward, right?

The guest tools correct time going backward.

Comment 7

[matthew zeier [:mrz]]

> 
> Did you try setting a clock= option to test if that fixes the problem?

When can I take bm-admin01 offline to add that?

Comment 8

[Justin Fitzhugh]

Already did this - reopen if clock is still off.

Comment 9

[J. Paul Reed [:preed]]

(In reply to comment #8)
> Already did this - reopen if clock is still off.

I checked this morning; setting clock=pit seemed to work. VM has been up for 5 days, and the clock is within a minute of what my laptop shows (my laptop is probably off; it doesn't use ntp).

So, it looks like the following process works:

1. Install NTP on the console in ESX.
2. Install Guest tools on all VMs; make sure timeSync="TRUE" is set in the .vmx file. If it's not, shutdown the VM, set it, and restart the VM. Make sure the Guest Tools run on startup.
3. For Linux VMs, set clock=pit on the kernel commandline if running ESX >= 3.0. For ESX 2.5.2, running the Guest Tools *seems* to be enough (we haven't had problems on the Build farm VMs).

Comment 10

[J. Paul Reed [:preed]]

Reopening to add clock=pit magic to dm-webtools01 and mpt-vpn.

Any other Linux VMs that need this?

Comment 11

[matthew zeier [:mrz]]

Re-closing - the other VMs are being tracked as downtime events.