533185 - Disable JIT debugger on windows slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Since http://hg.mozilla.org/projects/electrolysis/rev/a3fa339c9f61 landed the
"WINNT 5.2 electrolysis unit test" builder has been orange. It's timing out in
make check:
make[4]: Entering directory
`/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/ipdl'
.......................................................................
----------------------------------------------------------------------
Ran 71 tests in 22.047s

OK
make[4]: Leaving directory
`/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/ipdl'
make[4]: Entering directory
`/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/cxx'
make[5]: Entering directory
`/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/cxx/app'
make[5]: Nothing to be done for `check'.
make[5]: Leaving directory
`/e/builds/moz2_slave/electrolysis-win32-unittest/build/objdir/ipc/ipdl/test/cxx/app'
WARNING: RegisterWaitForSingleObject failed: 31: file
e:/builds/moz2_slave/electrolysis-win32-unittest/build/ipc/chromium/src/base/object_watcher.cc,
line 62
WARNING: Tried to RegisterCallback without an AtExitManager: file
e:/builds/moz2_slave/electrolysis-win32-unittest/build/ipc/chromium/src/base/at_exit.cc,
line 40
command timed out: 300 seconds without output

On the slave there is a dialog box up:
 nsAppShell:EventWIndow: mozilla-runtime.exe - Application Error
 The instruction at "0x7c81a379"" referenced memory at "0x0000009c" The memory
could not be "written".

Unusually, this also leads to buildbot dying and we lose a slave for the
electrolysis, tracemonkey, and mozila-1.9.1 pool.


Hrm, I thought the buildbot slaves had this dialog disabled at the Windows
level. This is done with System Properties/Advanced/Error Reporting: "Disable
Error Reporting" and uncheck "But notify me..."

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I'm trying this suggestion on moz2-win32-slave53. The dialog box goes on to say "Click OK to terminate, Cancel to debug" so it may be a different setting to change.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

With error reporting disabled, we got this
* mozilla-runtime.exe crashes with log like comment #0
* Windows displays an "application has been terminated" dialog, only option is OK button
* VNC server dead so connect to the console with VI
* shows same dialog as described in comment #0 and #1 (OK to terminate, Cancel to Debug)
* buildbot log has "Received SIGBREAK, shutting down" followed by it killing the call to "make check", some "lost remote" messages, and shutting down buildbot. This is the normal output from rebooting after each build.

So it looks like the debug prompt blocks the reboot. This is probably the first time a make check binary has started crashing since we enabled reboots after every build, so we may not have noticed this before.

moz2-win32-slave53 has error reporting re-enabled and returned to the pool.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #2)
> So it looks like the debug prompt blocks the reboot. 

Indeed it does. Deleting
 HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger
and changing 
 HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Auto
from 0 to 1 results in a clean reboot.

References:
http://msdn.microsoft.com/en-us/library/5hs4b7a6%28VS.80%29.aspx
http://msdn.microsoft.com/en-us/library/bb204634%28VS.85%29.aspx

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Exported AeDebug branch

For posterity.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: WIP

Installs OK, just having some issues with setting Debugger again on uninstall. The error message is:
 Sektion Registry_enable_jit (Kommando in Ziele 11):
 set "Debugger"="\"C:\\WINDOWS\\system32\\vsjitdebugger.exe\" -p %ld -e %ld"
 Char(s) at end of line not interpreted

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Could also delete VDebugger from when tried to do this originally
 https://wiki.mozilla.org/ReferencePlatforms/Win32#Version_9
but it's not obvious what winSt does if you try to delete a key that doesn't exist.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Working version

Compared to the WIP this also removes the VDebugger key, which was a failed attempt to do this previously, and has a mostly-correct uninstall. The Debugger key gets created with a REG_EXPAND_SZ type instead of REG_SZ. Since I tried to specifying REG_SZ in enable-jit.ins and still got REG_EXPAND_SZ I'm going to assume that windows insisted on that type and it's going to work like that.

I'm going to install this on the three staging slaves and check for really obvious bustage.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 421582 [details] [diff] [review]
Working version

http://hg.mozilla.org/build/opsi-package-sources/rev/710ba96db73b

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Set to install on the production win32-slave's - it'll get picked up as they complete a job and reboot. I'll check tomorrow for any that haven't got it yet and give them a kick.

All try-win32-slave* set to update. Still to do: 3 10 11 16 19-21 23-27 29

Win32 Ref platform VM updated (version 24) and docs updated. ix-ref not set to update; Ben, do we have a good place to track updating the new boxes ?

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

All current slaves updated. Leaving open for answer to question in comment #8.

Comment 11

[[e-mail] (:bhearsum)]

I just filed bug 541953 to track whatever delta there will be when these slaves arrive.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Thanks Ben.