509351 - Revive all RelEng machines/VMs that died when colo failed

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This bug is to track finding and fixing all of our machines/VMs that got hosed during the colo outage today.

Developers who see problems with infrastructure since early afternoon Sunday (9-aug-2009) should add details here so we can track and fix them.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

At this point I believe that Aki and I have brought up all the systems, but we know that there are a few holes in our nagios coverage.

A lot of the windows slaves for moz2 and try didn't come up cleanly. Do you have any more info on that Aki ?

Comment 2

[David Dahl :ddahl]

I noticed a build burning with this error: "Error: failed graph server post"

This was from http://hg.mozilla.org/mozilla-central/rev/bc6ffc794fc1

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1249907752.1249908654.12105.gz

Comment 3

[Chris Cooper [:coop] (he/him)]

Assigning to Alice who is on build duty this week so she can deal with any remaining fallout.(In reply to comment #2)


> I noticed a build burning with this error: "Error: failed graph server post"
> 
> This was from http://hg.mozilla.org/mozilla-central/rev/bc6ffc794fc1
> 
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1249907752.1249908654.12105.gz

Sounds like bug 493623.

Comment 4

[Aki Sasaki (not active)]

(In reply to comment #1)
> A lot of the windows slaves for moz2 and try didn't come up cleanly. Do you
> have any more info on that Aki ?

Not really.  They all came back with a reboot.
I do know that various "Do you want to notify Microsoft?" prompts kept killing my attempts at rebooting though.  Maybe those interfered with the buildbot batch script.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Seems like nothing else came up; ok to close this now?

Comment 6

[alice nodelman [:alice] [:anode]]

All is well.