682628 - Disable "JIT debugging" on Windows test slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Benoit Jacob [:bjacob] (mostly away)]

The abort() function is supposed to just terminate the process. But on certain Windows machines (apparently those that have the Visual Studio runtime installed?), it's overridden and results in a "Abort/Retry/Cancel" dialog popping up. Apparently "that's not a bug, that's a feature": http://msdn.microsoft.com/en-us/library/5hs4b7a6%28v=VS.100%29.aspx

The Windows (both XP and 7) test slaves seem to have this behavior.

The abort() function is something that we can't completely avoid calling: even if we never used it in our codebase, we still use third-party software that calls it. It's called by the standard assert() macro.

This is a major pain on test slaves for the following reason, see bug 679864 comment 23 for the concrete story. Suppose that as part of mochitest-1 you have a test that, due to a bug, results in a abort() call; and then, later in mochitest-1, you have another test that relies on the Mochitest/Firefox window having focus. The latter test will then fail because the dialog window popped up during the former test stole the focus.

To avoid hard to debug problems like that in the future, it would be nice to get rid of these pop-up dialogs altogether on the Windows test slaves.

This MSDN page gives instructions on how to disable this, by setting a registry key:
http://msdn.microsoft.com/en-us/library/5hs4b7a6%28v=VS.100%29.aspx

Comment 1

[Armen [:armenzg]]

We have this disable for win32 and win64 slaves.

For win64 we remove some registry entries and did the trick:
https://wiki.mozilla.org/ReferencePlatforms/Win64#Disable_JIT_debugger

This is important to get done but I don't yet have time for this.

Comment 2

[Chris Cooper [:coop] (he/him)]

Yes, as Armen indicates, I thought we already did this. I guess we'll need to sweep through the Windows pool to be sure.

Comment 3

[Benoit Jacob [:bjacob] (mostly away)]

The issue (or some slightly different issue?) must still be present on many Windows test slaves, as I've been hitting it consistently when trying to land bug 679864.

For example:
http://tbpl.allizom.org/php/getParsedLog.php?id=6061835&full=1

builder: try_win7-debug_test-mochitests-1
slave: talos-r3-w7-008
starttime: 1313986233.37

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Chris Cooper [:coop] from comment #2)
> Yes, as Armen indicates, I thought we already did this. I guess we'll need
> to sweep through the Windows pool to be sure.

We definitely did on the builders, but the requirement for testers probably came along when bug 562459 deployed the debug CRT.

Comment 5

[Chris Cooper [:coop] (he/him)]

Going to start iterating through slaves this afternoon.

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #1)
> We have this disable for win32 and win64 slaves.
> 
> For win64 we remove some registry entries and did the trick:
> https://wiki.mozilla.org/ReferencePlatforms/Win64#Disable_JIT_debugger
> 
> This is important to get done but I don't yet have time for this.

Armen: can I take away from this that the Win64 test slaves won't have this problem, i.e. you've already removed the registry entries on them (and the ref image)?

Comment 7

[Chris Cooper [:coop] (he/him)]

xp slaves are all done, save talos-r3-xp-045 (bug 661377).

FWIW, I'm running the following commands via csshX to get this done:

reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug" /v Debugger /f
reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug"
reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework" /v DbgManagedDebugger /f
reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework"

I'll tackle Win7 tomorrow.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #7)
> I'll tackle Win7 tomorrow.

I've gone through all the Win7 test slaves now, but those registry keys didn't exist anywhere. Just as well too, because deleting the keys from the command-line didn't seem tenable despite several attempts using runas and subinacl.

Filed bug 685879 on talos-r3-w7-036 which was down (again).

(In reply to Benoit Jacob [:bjacob] from comment #0) 
> The Windows (both XP and 7) test slaves seem to have this behavior.

Was there a particular win7 machine that was demonstrating this behaviour?

Comment 9

[Chris Cooper [:coop] (he/him)]

Armen told me the w764 test slaves were already done, and I double-checked them to be sure.

Updating RefPlatform docs now.

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #9)
> Armen told me the w764 test slaves were already done, and I double-checked
> them to be sure.

Armen just clarified that he was only talking about w64 slaves, so it's a good thing I double-checked!

Comment 11

[Chris Cooper [:coop] (he/him)]

Attachment: Delete registry keys for JIT debugging via OPSI

Comment 12

[Armen [:armenzg]]

Comment on attachment 559505 [details] [diff] [review]
Delete registry keys for JIT debugging via OPSI

Looks good.

Should we adjust "Message" as well? Not sure. Don't care enough.

Comment 13

[Chris Cooper [:coop] (he/him)]

Comment on attachment 559505 [details] [diff] [review]
Delete registry keys for JIT debugging via OPSI

http://hg.mozilla.org/build/opsi-package-sources/rev/1f0f166cab4b

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #9) 
> Updating RefPlatform docs now.

Done.

(In reply to Chris Cooper [:coop] from comment #13)
> Comment on attachment 559505 [details] [diff] [review]
> Delete registry keys for JIT debugging via OPSI

This is in production-opsi now, but the keys have already been removed from the slaves, so there's nothing left to do here.