427248 - most trunk tinderboxes red or orange

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Josh Aas]

The tree closed late last night (around 4 AM April 5th) for a single orange on each platform, now almost every box has issues. Doesn't seem to be related to code or tests, we need to get the tree open again.

Comment 1

[Mike Connor [:mconnor]]

We're three days from planned code freeze, lets get this rolling, it'll already take hours to clear.

Comment 2

[chizu]

Some of these have been restarted where that looked like it might fix things, and some of the others made it past the failure on their own this time around. No idea what caused them to all die at once.

Still working on getting the last few going.

Comment 3

[Jeff Walden [:Waldo]]

The PGO box is failing due to a test timeout midway through Mochitests:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1207373279.1207382897.4678.gz

I have a suspicion this might be due to bug 418009, where somehow the server is being pushed into making too much output.  There's a patch in the bug which could be used to test this hypothesis awaiting review now.

That said, the timeout code that kills runaway Mochitest is clearly doing something wrong and needs to be fixed, but if we can fix the immediate problem that problem is a bit less pressing.

Comment 4

[chizu]

All the failures from this morning are fixed. qm-centos5-01 and qm-xserve01 are still orange with one test failure, bug 425987.

bm-win2k3-pgo01 problem is bug 426997.