878809 - Urgent - Please update the timezone to be PDT

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Armen [:armenzg]]

It seems that the our XP machines are either using the wrong time or the wrong timezone and/or something updated on Friday.

This is causing a bunch of failures in our test infrastructure.

The sooner this gets fixed, the easier it will be for sheriffs and devs.

On another note, can I please get some instructions on how to login as root on those machines? I couldn't switch users with VNC and RDP was not doing anything.

Comment 1

[Phil Ringnalda (:philor)]

If it helps with diagnosis, it doesn't need to have been a change on Friday, because the test failure only happens when there are a particular number of hours to the next Daylight/Standard time changeover, so it could have happened up to a month before and been dormant (or we could have not been running tests a month ago, don't remember).

Comment 2

[Van Le [:van]]

Do you need DCOPs to manually connect a crash cart to each host and reconfigure or is this something that can be scripted or configured by puppet? 

If you do need hands on, can you provide a list of hosts that need attention?

Comment 3

[Van Le [:van]]

err, weird mid air collision. Thanks for clearing it up Amy.

Comment 4

[Amy Rich [:arr] [:arich]]

This is not a DCOps bug.

Armen: Can you please specify if this is the iX machines or the r3 minis?

Comment 5

[Amy Rich [:arr] [:arich]]

This is impacting the iX machines only.  Suspicion is that we don't have the DST patch installed so the machines are still living in the dark ages when we changed over to DST and back on a normal schedule.  Mark, can you please take a look and verify/fix?

Comment 6

[Q]

The TZ patch is in place on sp3. This is a combination of issues I will document in a bit. I am sending out a fix for all testers in the domain to ensure this doesn't happen again or in another OS.

Comment 7

[Q]

Looks like all documented machines have been fixed via a combination of registry pushes and a immediate scheduled task. Can we get some releng verification that things look good?

Comment 8

[Phil Ringnalda (:philor)]

Just by comparing timestamps that come from the master to timestamps that come from the slave, one I looked at that started at 12:15 hadn't been fixed yet, all three that started at 12:25 had (bug 630428 comment 8 tells how to calculate the times and days when the jsreftest failure that pointed out the problem to us will next hit, since it only happens during three one-hour periods on certain days, but it makes my head hurt so I don't actually know whether there's another episode of that coming up today to verify, or there won't be another for a month now).

Comment 9

[Armen [:armenzg]]

(In reply to Q from comment #7)
> Looks like all documented machines have been fixed via a combination of
> registry pushes and a immediate scheduled task. Can we get some releng
> verification that things look good?

I can verify that t-xp32-ix-056 which was 2 hours behind my time zone and it is now 3 hours behind.

I say we're fixed for now. As philor says, we might not been able to know for a while. We could then re-open if needed.

So far I haven't seen more reports of the problem happening.

When you have a chance could you also be able to email me this info? Please :)
(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) from comment #0)
> On another note, can I please get some instructions on how to login as root
> on those machines? I couldn't switch users with VNC and RDP was not doing
> anything.