I'm filing this at 6:36, Hudson thinks it's 5:18:51 PM.
Hudson thinks it's Dec 30, 2009 9:29:03 PM right now
This makes me sad.
/etc/ntp/step-tickers was empty. This is supposed to get set by kickstart. shyam, any clue why that file was empty on this host? Was it built before you added that fix? fyi, I've gone and fixed the clock on it.
mrz - yeah, this was the old sm-mozid01 and was built WAY before I made that fix. (In reply to comment #3) > This makes me sad. I'm sad that you're sad. First sad event of 2010 :( boohoo!
Looks like it's losing about 2 minutes every 15 minutes. Current time is 18 minutes slow.
Can I halt this VM any time to check the timesync options?
Taking it down this weekend would be fine
meaning, anytime between now and monday. If you need to do it at another time we should take downtime so devs aren't expecting it to work.
Time sync turn on, verified time on host.
Is this thing on? "Firefox can't establish a connection to the server at sm-hudson01.mozilla.org:8080."
Nope - [root@sm-hudson01 ~]# service httpd status httpd is stopped [root@sm-hudson01 ~]# service httpd start Starting httpd: [ OK ] [root@sm-hudson01 ~]# chkconfig httpd on [root@sm-hudson01 ~]#
Hudson, since it is made of hate and java, runs its own web server on 8080. It looks like you have to kick that one separately. http://sm-hudson01.mozilla.org:8080/job/addons.mozilla.org/ is still unavailable.
1. clock keeps drifting. I'm out of ideas. Shyam? 2. So kill apache? [root@sm-hudson01 init.d]# service hudson status hudson dead but pid file exists [root@sm-hudson01 init.d]# rm /var/run/hudson.pid rm: remove regular file `/var/run/hudson.pid'? yes [root@sm-hudson01 init.d]# service hudson start Starting Hudson [ OK ]
1. No clue, will poke later. It's only behind by 2 mins. Probably something with VMWare :| 2. Is fixed. For some stupid reason, /var/log/hudson/hudson.log was 644 owner and group root and so hudson would just die without any error and never start up, which is fail. I've fixed the perms and it should be fine now.
For the sake of completeness : dev-vmware02 which is the host sm-hudson01 was running on didn't have it's time set right (was behind by 2 mins) and the VMs pick up their CMOS time from their ESX hosts. Since this was behind by 2 mins, the VMs were all behind by the same amount of time. Two methods of time syncs on VMs - 1) OS side (ntp etc) 2) VMWare Tools. We've gone with the latter here at Mozilla and that's what we'll stick to. Now that the CMOS time for the machines is fixed, a restart of VMWare Tools on the host brought the time in sync and it's stayed that way for the last 20 mins or so. For the curious, more info at http://www.vmware.com/pdf/vmware_timekeeping.pdf and for the others, thanks for watching! show's over, move along now.