sm-hudson01 does not have the right time

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
--
minor
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: jbalogh, Assigned: fox2mike)

Tracking

Details

(Reporter)

Description

8 years ago
I'm filing this at 6:36, Hudson thinks it's 5:18:51 PM.
(Assignee)

Comment 1

8 years ago
Fixed.
Assignee: server-ops → shyam
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Hudson thinks it's Dec 30, 2009 9:29:03 PM right now
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
This makes me sad.

Comment 4

8 years ago
/etc/ntp/step-tickers was empty.

This is supposed to get set by kickstart.  

shyam, any clue why that file was empty on this host?  Was it built before you added that fix?

fyi, I've gone and fixed the clock on it.
(Assignee)

Comment 5

8 years ago
mrz - yeah, this was the old sm-mozid01 and was built WAY before I made that fix.

(In reply to comment #3)
> This makes me sad.

I'm sad that you're sad. First sad event of 2010 :( boohoo!
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
Looks like it's losing about 2 minutes every 15 minutes.  Current time is 18 minutes slow.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 7

8 years ago
Can I halt this VM any time to check the timesync options?
Taking it down this weekend would be fine
meaning, anytime between now and monday.  If you need to do it at another time we should take downtime so devs aren't expecting it to work.
Time sync turn on, verified time on host.
(Reporter)

Comment 11

8 years ago
Is this thing on? "Firefox can't establish a connection to the server at sm-hudson01.mozilla.org:8080."
Nope - 

[root@sm-hudson01 ~]# service httpd status
httpd is stopped
[root@sm-hudson01 ~]# service httpd start
Starting httpd:                                            [  OK  ]
[root@sm-hudson01 ~]# chkconfig httpd on
[root@sm-hudson01 ~]#
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
(Reporter)

Comment 13

8 years ago
Hudson, since it is made of hate and java, runs its own web server on 8080.  It looks like you have to kick that one separately. http://sm-hudson01.mozilla.org:8080/job/addons.mozilla.org/ is still unavailable.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
1. clock keeps drifting.  I'm out of ideas.  Shyam?

2. So kill apache?

[root@sm-hudson01 init.d]# service hudson status
hudson dead but pid file exists

[root@sm-hudson01 init.d]# rm /var/run/hudson.pid 
rm: remove regular file `/var/run/hudson.pid'? yes
[root@sm-hudson01 init.d]# service hudson start
Starting Hudson                                            [  OK  ]
(Assignee)

Comment 15

8 years ago
1. No clue, will poke later. It's only behind by 2 mins. Probably something with VMWare :|

2. Is fixed. For some stupid reason, /var/log/hudson/hudson.log was 644 owner and group root and so hudson would just die without any error and never start up, which is fail. I've fixed the perms and it should be fine now.
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
(Assignee)

Comment 16

8 years ago
For the sake of completeness :

dev-vmware02 which is the host sm-hudson01 was running on didn't have it's time set right (was behind by 2 mins) and the VMs pick up their CMOS time from their ESX hosts.

Since this was behind by 2 mins, the VMs were all behind by the same amount of time.

Two methods of time syncs on VMs - 1) OS side (ntp etc) 2) VMWare Tools. We've gone with the latter here at Mozilla and that's what we'll stick to. Now that the CMOS time for the machines is fixed, a restart of VMWare Tools on the host brought the time in sync and it's stayed that way for the last 20 mins or so.

For the curious, more info at http://www.vmware.com/pdf/vmware_timekeeping.pdf and for the others, thanks for watching! show's over, move along now.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.