535718 - sm-hudson01 does not have the right time

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Jeff Balogh (:jbalogh)]

I'm filing this at 6:36, Hudson thinks it's 5:18:51 PM.

Comment 1

[Shyam Mani [:fox2mike]]

Fixed.

Comment 2

[Wil Clouser [:clouserw]]

Hudson thinks it's Dec 30, 2009 9:29:03 PM right now

Comment 3

[Michael Morgan [:morgamic]]

This makes me sad.

Comment 4

[matthew zeier [:mrz]]

/etc/ntp/step-tickers was empty.

This is supposed to get set by kickstart.  

shyam, any clue why that file was empty on this host?  Was it built before you added that fix?

fyi, I've gone and fixed the clock on it.

Comment 5

[Shyam Mani [:fox2mike]]

mrz - yeah, this was the old sm-mozid01 and was built WAY before I made that fix.

(In reply to comment #3)
> This makes me sad.

I'm sad that you're sad. First sad event of 2010 :( boohoo!

Comment 6

[Wil Clouser [:clouserw]]

Looks like it's losing about 2 minutes every 15 minutes.  Current time is 18 minutes slow.

Comment 7

[matthew zeier [:mrz]]

Can I halt this VM any time to check the timesync options?

Comment 8

[Wil Clouser [:clouserw]]

Taking it down this weekend would be fine

Comment 9

[Wil Clouser [:clouserw]]

meaning, anytime between now and monday.  If you need to do it at another time we should take downtime so devs aren't expecting it to work.

Comment 10

[matthew zeier [:mrz]]

Time sync turn on, verified time on host.

Comment 11

[Jeff Balogh (:jbalogh)]

Is this thing on? "Firefox can't establish a connection to the server at sm-hudson01.mozilla.org:8080."

Comment 12

[matthew zeier [:mrz]]

Nope - 

[root@sm-hudson01 ~]# service httpd status
httpd is stopped
[root@sm-hudson01 ~]# service httpd start
Starting httpd:                                            [  OK  ]
[root@sm-hudson01 ~]# chkconfig httpd on
[root@sm-hudson01 ~]#

Comment 13

[Jeff Balogh (:jbalogh)]

Hudson, since it is made of hate and java, runs its own web server on 8080.  It looks like you have to kick that one separately. http://sm-hudson01.mozilla.org:8080/job/addons.mozilla.org/ is still unavailable.

Comment 14

[matthew zeier [:mrz]]

1. clock keeps drifting.  I'm out of ideas.  Shyam?

2. So kill apache?

[root@sm-hudson01 init.d]# service hudson status
hudson dead but pid file exists

[root@sm-hudson01 init.d]# rm /var/run/hudson.pid 
rm: remove regular file `/var/run/hudson.pid'? yes
[root@sm-hudson01 init.d]# service hudson start
Starting Hudson                                            [  OK  ]

Comment 15

[Shyam Mani [:fox2mike]]

1. No clue, will poke later. It's only behind by 2 mins. Probably something with VMWare :|

2. Is fixed. For some stupid reason, /var/log/hudson/hudson.log was 644 owner and group root and so hudson would just die without any error and never start up, which is fail. I've fixed the perms and it should be fine now.

Comment 16

[Shyam Mani [:fox2mike]]

For the sake of completeness :

dev-vmware02 which is the host sm-hudson01 was running on didn't have it's time set right (was behind by 2 mins) and the VMs pick up their CMOS time from their ESX hosts.

Since this was behind by 2 mins, the VMs were all behind by the same amount of time.

Two methods of time syncs on VMs - 1) OS side (ntp etc) 2) VMWare Tools. We've gone with the latter here at Mozilla and that's what we'll stick to. Now that the CMOS time for the machines is fixed, a restart of VMWare Tools on the host brought the time in sync and it's stayed that way for the last 20 mins or so.

For the curious, more info at http://www.vmware.com/pdf/vmware_timekeeping.pdf and for the others, thanks for watching! show's over, move along now.