1683177 - TypeError: TerminateProcess() argument 1 must be int, not AutoHANDLE

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Robin Steuber (she/her) [:bytesized]]

Recently, when I run mochitests on Windows, I consistently see this traceback in the output:

> ./mach mochitest toolkit/mozapps/update/tests/browser/browser_doorhanger_bc_patch_completeBadSize.js
...
Traceback (most recent call last):
  File "c:/Users/bytesized/mozilla-central/obj-unoptimized\_tests\testing\mochitest\runtests.py", line 614, in stop
    self._process.proc.terminate()
  File "c:\mozilla-build\python3\lib\subprocess.py", line 1314, in terminate
    _winapi.TerminateProcess(self._handle, 1)
TypeError: TerminateProcess() argument 1 must be int, not AutoHANDLE
...

The traceback appears on stderr rather than stdout like the rest of the test output, so it's much easier to see if you redirect stdout elsewhere:

> ./mach mochitest toolkit/mozapps/update/tests/browser/browser_doorhanger_bc_patch_completeBadSize.js > /dev/null

Comment 1

[Geoff Brown [:gbrown]]

I see that also, if I try to run mochitest with py3 in CI:

https://treeherder.mozilla.org/logviewer?job_id=331396058&repo=try&lineNumber=2007

Comment 2

[Geoff Brown [:gbrown]]

Attachment: Bug 1683177 - Improve mochitest web server shutdown and avoid exception under py3 on Windows; r=

The exception seen in this bug can be avoided by simply replacing the terminate() call
with kill(). While we're here, I've tried to simplify the shutdown logic and improve
logging.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1f4840599e64
Improve mochitest web server shutdown and avoid exception under py3 on Windows; r=ahal

Comment 4

[Sandor Molnar[:smolnar]]

Backed out for causing mochitest failures on windows in quota/QuotaManagerService

Backout link: https://hg.mozilla.org/integration/autoland/rev/e6b1cff8631509a6bc70a6bd6fd8d48e6c115861

Push with failures

Failure log

INFO - Stopping web server
[task 2021-03-08T21:50:15.052Z] 21:50:15 INFO - Server shut down.
[task 2021-03-08T21:50:15.294Z] 21:50:15 INFO - Assertion failure: !mIdleObserverRegistered, at /builds/worker/checkouts/gecko/dom/quota/QuotaManagerService.cpp:196
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #01: soundtouch::SoundTouch::operator=[Z:\task_1615237849\build\application\firefox\xul.dll +0x46c44df]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #02: soundtouch::SoundTouch::operator=[Z:\task_1615237849\build\application\firefox\xul.dll +0x46c440b]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #03: Ordinal0[Z:\task_1615237849\build\application\firefox\xul.dll +0x19eaf0]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #04: soundtouch::SoundTouch::operator=[Z:\task_1615237849\build\application\firefox\xul.dll +0x46c461d]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #05: Ordinal0[Z:\task_1615237849\build\application\firefox\xul.dll +0x19eabe]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #06: Ordinal0[Z:\task_1615237849\build\application\firefox\xul.dll +0x18d358]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #07: Ordinal0[Z:\task_1615237849\build\application\firefox\xul.dll +0x1a9e35]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #08: Ordinal0[Z:\task_1615237849\build\application\firefox\xul.dll +0x22b5e4]
[task 2021-03-08T21:50:15.364Z] 21:50:15 INFO - #09: Ordinal0[Z:\task_1615237849\build\application\firefox\xul.dll +0x1218a29]
[task 2021-03-08T21:50:15.365Z] 21:50:15 INFO - #10: Ordinal0[Z:\task_1615237849\build\tests\bin\xpcshell.exe +0x1297]
[task 2021-03-08T21:50:15.365Z] 21:50:15 INFO - #11: Ordinal0[Z:\task_1615237849\build\tests\bin\xpcshell.exe +0x11a7]
[task 2021-03-08T21:50:15.365Z] 21:50:15 INFO - #12: TargetNtUnmapViewOfSection[Z:\task_1615237849\build\tests\bin\xpcshell.exe +0x7d508]
[task 2021-03-08T21:50:15.365Z] 21:50:15 INFO - #13: BaseThreadInitThunk[C:\Windows\System32\KERNEL32.DLL +0x13034]
[task 2021-03-08T21:50:15.365Z] 21:50:15 INFO - #14: RtlUserThreadStart[C:\Windows\SYSTEM32\ntdll.dll +0x71461]
[task 2021-03-08T21:50:15.524Z] 21:50:15 INFO - Stopping web socket server
[task 2021-03-08T21:50:15.544Z] 21:50:15 INFO - Stopping ssltunnel

Comment 5

[Geoff Brown [:gbrown]]

I verified this problem on try:
https://treeherder.mozilla.org/jobs?repo=try&revision=fc47c2b4349586277f202b8f60fe326b6b03b09e
vs
https://treeherder.mozilla.org/jobs?repo=try&revision=ed9a1e29fae366c271b55a2a16aeed012840e0c7

With my patch, it appears that the web server xpcshell intermittently crashes, on Windows only, after being told to shutdown via the shutdown url, but before any attempt is made to kill it. From my patch's perspective, I suppose this happens during the new wait() call. But surely wait() does not affect the process in a way that could cause it to crash? In the existing code, is the crash avoided by the immediate termination? Investigating...

Comment 6

[Geoff Brown [:gbrown]]

Using mozprocess wait() with a timeout parameter was a bad idea. subprocess wait()'s timeout was added in python 3.3, and is inconsistently implemented in mozprocess.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3165f2868244
Improve mochitest web server shutdown and avoid exception under py3 on Windows; r=ahal

Comment 8

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/3165f2868244