536587 - random orange: test suites hit timeout after apparently succesfully completing run

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P3)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I think I've seen this random orange happen before (although I'm not sure), and I can't find a bug on it, so I'm filing it.

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1261570572.1261578399.387.gz
WINNT 5.2 mozilla-central debug test everythingelse on 2009/12/23 04:16:12  

was a unit test run in which xpcshell, jsreftest, and mochitest-ipcplugins all timed out in the same unusual way.  The test completed, with the final TinderboxPrint: of the test's output, and *then* the test timed out:

program finished with exit code 0
elapsedTime=1987.375000
TinderboxPrint: xpcshell<br/>724/0
buildbot.slave.commands.TimeoutError: command timed out: 3600 seconds elapsed
TinderboxPrint: xpcshell<br/><em class="testfail">timeout</em>

...

program finished with exit code 0
elapsedTime=2093.446000
TinderboxPrint: jsreftest<br/>53963/0/908
buildbot.slave.commands.TimeoutError: command timed out: 3600 seconds elapsed
TinderboxPrint: jsreftest<br/><em class="testfail">timeout</em>

...

program finished with exit code 0
elapsedTime=52.328000
TinderboxPrint: mochitest-ipcplugins<br/>212/0/0
buildbot.slave.commands.TimeoutError: command timed out: 5400 seconds elapsed
TinderboxPrint: mochitest-ipcplugins<br/><em class="testfail">timeout</em>

Comment 1

[Chris AtLee [:catlee]]

Haven't seen this one before.  Looks like a buildbot bug in buildbot/slave/commands.py in the finished method maybe...The maxTimer isn't being canceled properly?

If you see this again, please post a link to the log here so we can do more analysis.

Comment 2

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 3

[Uli Link (:ul-mcamafia)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1272196746.1272198704.8357.gz

Comment 4

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1274896302.1274898415.26432.gz#err1

Comment 5

[Daniel Holbert [:dholbert]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1275679227.1275681109.24053.gz
WINNT 5.2 mozilla-central debug test mochitests-1/5 on 2010/06/04 12:20:27
s: win32-slave37
{
64806 INFO SimpleTest FINISHED
[...]
TEST-UNEXPECTED-FAIL | automation.py | application timed out after 330 seconds with no output
[...]
PROCESS-CRASH | automation.py | application crashed (minidump found)
Operating system: Windows NT
                  5.2.3790 Service Pack 2
CPU: x86
     GenuineIntel family 6 model 23 stepping 8
     1 CPU

Crash reason:  EXCEPTION_ACCESS_VIOLATION
Crash address: 0x0
}

Comment 6

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1275693404.1275695644.27653.gz
WINNT 5.2 mozilla-central debug test mochitests-1/5 on 2010/06/04 16:16:44
s: win32-slave14

Feels a bit suspicious, suddenly seeing two of these in one day.

Comment 7

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1276322460.1276324174.19036.gz
WINNT 5.2 mozilla-central debug test mochitests-1/5 on 2010/06/11 23:01:00
s: win32-slave31

Comment 8

[Chris Cooper [:coop] (he/him)]

I *think* this may have been fixed by bug 571619, but the tinderbox logs have since expired so there's nothing for me to work with here, sadly.

Please reopen if this recurs.