533141 - Electrolysis bustage leading to disconnected build slaves

Status: RESOLVED DUPLICATE of bug 532983

(Core :: IPC, defect)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Hi Chris, Benjamin,

Since http://hg.mozilla.org/projects/electrolysis/rev/a3fa339c9f61 landed the "WINNT 5.2 electrolysis unit test" builder has been orange. It's timing out in make check:
make[4]: Entering directory `/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/ipdl'
.......................................................................
----------------------------------------------------------------------
Ran 71 tests in 22.047s

OK
make[4]: Leaving directory `/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/ipdl'
make[4]: Entering directory `/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/cxx'
make[5]: Entering directory `/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/cxx/app'
make[5]: Nothing to be done for `check'.
make[5]: Leaving directory `/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/cxx/app'
WARNING: RegisterWaitForSingleObject failed: 31: file e:/builds/moz2_slave/electrolysis-win32-unittest/build/ipc/chromium/src/base/object_watcher.cc, line 62
WARNING: Tried to RegisterCallback without an AtExitManager: file e:/builds/moz2_slave/electrolysis-win32-unittest/build/ipc/chromium/src/base/at_exit.cc, line 40
command timed out: 300 seconds without output

On the slave there is a dialog box up:
 nsAppShell:EventWIndow: mozilla-runtime.exe - Application Error
 The instruction at "0x7c81a379"" referenced memory at "0x0000009c" The memory could not be "written".

Unusually, this also leads to buildbot dying and we lose a slave for the electrolysis, tracemonkey, and mozila-1.9.1 pool. Could you please look into a fix ?

Cheers,
Nick

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Hrm, I thought the buildbot slaves had this dialog disabled at the Windows level. This is done with System Properties/Advanced/Error Reporting: "Disable Error Reporting" and uncheck "But notify me..."

I was going to say that my push earlier today http://hg.mozilla.org/projects/electrolysis/rev/51d0aea0390b would fix this, but this `make check` error appears to be unrelated to the orange in mochitest-ipcplugins which I was examining. I don't have access to a tree which can back this out until I arrive in CA tomorrow evening, so I hope that cjones or bent can look at it.

--BDS

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Raising priority. Each time this happens, it removes a slave from the pool, which impacts all win32 builds across electrolysis, tracemonkey, and mozila-1.9.1 branches.

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Spun off bug 533185 for the infra issue.