480322 - Linux unittest box having massive timeouts, crashing -- needs attention probably

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Daniel Holbert [:dholbert]]

The linux unittest box just went wonky for two consecutive cycles.

First cycle:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1235645906.1235650452.6081.gz
Description:
 - Massive timeouts in reftests
 - Crash during crashtests (after loading 323495-1.html)
 - Hang while starting mochitests
 - Timeout in mochichrome test test_domstorage.xul

Second cycle:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1235653106.1235659476.530.gz
Description:
 - "configure: line 2911: echo: write error: Broken pipe"
 - Segfault in crashtests after loading 354458-1.html
 - Timeout in mochitests (300sec) after passing one of the tests in test_bug467672-2.html
 - Mochichrome failures in test_bug418874.xul

The only code change between these two orange builds and the previous green build was the addition of two crashtests, which both passed, so I don't think that triggered the problem.  That changeset is: http://hg.mozilla.org/mozilla-central/rev/d3ce1f2c44bb

Comment 1

[[e-mail] (:bhearsum)]

For the record, there are 16 different slaves that do these builds. If there's a machine problem it's affecting most or all of them. I'm still investigating this a bit and won't be rebooting anything just yet.

Comment 2

[[e-mail] (:bhearsum)]

So, the two latest oranges are on moz2-linux-slave18 and moz2-linux-slave07 respectively. slave18 is freshly cloned, and this was it's first m-c unittest build. slave07 has been around awhile, and did a dep build. Given that, I can pretty safely say that clobbering won't help.

I logged on to both of the machines and there weren't any hung processes, Xvfb and metacity were both started properly - everything looked fine there.

I also checked out the load on the ESX hosts - it was fine.

I'd like to see another run on one of these machines with the same revision before doing any rebooting. If it's an intermittent test failure this should help determine that. If we get the same results again we can reboot one of them and see if that helps.

Both are building right now, as soon as one becomes free I'll pull it out of the pool and run the tests manually on it - since there's no way to force a run on a specific slave.

Comment 3

[Daniel Holbert [:dholbert]]

(In reply to comment #2)
> slave18 is freshly cloned, and this was it's first m-c unittest
> build.

Interesting -- that's the one that had the "configure: line 2911: echo: write error: Broken pipe".  Could that be related to it being freshly cloned?  (something not getting set up correctly, maybe?)

In any case, thanks for looking into this!

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> (In reply to comment #2)
> > slave18 is freshly cloned, and this was it's first m-c unittest
> > build.
> 
> Interesting -- that's the one that had the "configure: line 2911: echo: write
> error: Broken pipe".  Could that be related to it being freshly cloned? 
> (something not getting set up correctly, maybe?)

It's possible. However, it's done dep and leak test builds and not had that problem - so I dunno!

Comment 5

[Daniel Holbert [:dholbert]]

(In reply to comment #0)
>  - Crash during crashtests (after loading 323495-1.html)
[ snip ]
>  - Segfault in crashtests after loading 354458-1.html

Bug 480300 covers ths sporadic-crash-during-crashtests issue.

Comment 6

[[e-mail] (:bhearsum)]

moz2-linux-slave07 became idle. I pulled it out of the pool and am running through the tests manually right now.

Comment 7

[[e-mail] (:bhearsum)]

I just repro'ed the crashtest crash on slave07 after about 5 runs. Unfortunately, I forgot to set XPCOM_DEBUG_BREAK, so I'm trying to repro it again now.

Comment 8

[[e-mail] (:bhearsum)]

I suspect the mochitest timeouts are caused by the crashtest crash filed in https://bugzilla.mozilla.org/show_bug.cgi?id=480300. I'll keep this bug open until that fix lands though, to be sure. Please let me know if you see a mochitest timeout which isn't preceeded by a crashtest crash.

Comment 9

[[e-mail] (:bhearsum)]

I'm pretty certain this has been fixed since http://hg.mozilla.org/mozilla-central/rev/87f92525dccf.