1753555 - _run_python_test hangs for 3 minutes (inside ProcessHandler) on Windows

Status: RESOLVED FIXED

(Testing :: Python Test, defect, P3)

Description

[Alex Hochheiden [:ahochheiden] [he/him]]

While running tests for mozversioncontrol (via ./mach python-test --subsuite=mozversioncontrol) on my Windows machine I noticed that the tests took much longer to run than they should (Over 3 minutes, instead of ~20 seconds as they do on my Linux VM). There were also a bunch of warnings that appeared when the tests finally did complete:

WARNING | IO Completion Port failed to signal process shutdown
WARNING | IO Completion Port failed to signal process shutdown
Parent process 21640 exited with children alive:
PIDS: 18040
Parent process 10644 exited with children alive:
PIDS: 23140
Attempting to kill them, but no guarantee of success
Attempting to kill them, but no guarantee of success

I did a little bit of investigating and found the cause of the 3 minute hang here: https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/processhandler.py#572-672

It seems like the child process for each tests isn't signalling the parent upon test completion. I wasn't able to figure out the root cause of that, but I came up with a workaround that just replaces ProcessHandler in _run_python_test with a more standard approach that yields the same results, just much quicker.

I'll raise a different bug to investigate/solve the problems with ProcessHandler, I just wanted to raise a bug here so I can push this quick low effort/low risk fix which will save a bunch of time on testing on Windows.

Comment 1

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1753555 - Replaced `ProcessHandler` in `run_python_test` with a more standard solution r?ahal

There's a bug with ProcessHandler on Windows that sometimes causes
tests to hang for an additional 3 minutes. This is a workaround that
just replaces ProcessHandler with a standard Python3 approach that
achieves the same thing.

Also added some logging output as the threads are created as some
additional feedback. Prior to this, the tests silently started
running without any indication.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/22d204407826
Replaced `ProcessHandler` in `run_python_test` with a more standard solution r=ahal

Comment 3

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/22d204407826

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Alex Hochheiden [:ahochheiden] from comment #0)

I'll raise a different bug to investigate/solve the problems with ProcessHandler, I just wanted to raise a bug here so I can push this quick low effort/low risk fix which will save a bunch of time on testing on Windows.

fwiw, I think you did exactly the right thing here, going with subprocess. mozprocess has a ton of special-purpose code to work around a variety of legacy issues. No one know if any of that is useful any more, and we do know that mozprocess is buggy, especially on Windows. Any time mozprocess causes us trouble, I advocate for switching to subprocess.