[mozprocess] mozprocess ProcessReader waits for stdout/stderr threads after timeout

NEW
Unassigned

Status

Testing
Mozbase
2 months ago
6 days ago

People

(Reporter: gbrown, Unassigned)

Tracking

(Blocks: 3 bugs)

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 months ago
I encountered this while investigating current mochitest hangs. Consider the case where an output timeout has triggered a break from the loop at

https://dxr.mozilla.org/mozilla-central/rev/cad9c9573579698c223b4b6cb53ca723cd930ad2/testing/mozbase/mozprocess/mozprocess/processhandler.py#980

Now the ProcessReader continues to wait for the stdout/stderr reader threads to end:

https://dxr.mozilla.org/mozilla-central/rev/cad9c9573579698c223b4b6cb53ca723cd930ad2/testing/mozbase/mozprocess/mozprocess/processhandler.py#996-999

I think those threads normally end because the test harness output timeout handler has killed the process (typically via mozcrash or similar killAndGetStack()), causing readline() to return None:

https://dxr.mozilla.org/mozilla-central/rev/cad9c9573579698c223b4b6cb53ca723cd930ad2/testing/mozbase/mozprocess/mozprocess/processhandler.py#933-935

But what if the process didn't die (bug 1404190?), and the stream remains open? The ProcessReader knows it has timed out, has drained its queue and isn't interested in any more output, but it still waits for those threads...possibly forever.
(Reporter)

Updated

2 months ago
See Also: → bug 1421295
Blocks: 1407383
Blocks: 1391545
(Reporter)

Comment 1

2 months ago
I don't know how to proceed here. The existing code is efficient and straight forward, blocking on readline(). It looks to be very difficult to interrupt a thread in python...and probably dangerous. Other than killing the process (since we are having trouble with that), is there a way to break out of readline()? Close the stream??

asyncio could be helpful...but it is only available in python 3, afaik.
(Reporter)

Comment 2

2 months ago
(In reply to Geoff Brown [:gbrown] from comment #1)
> Close the stream??

I tried closing stdout/stderr from another thread (threading.Timer), but it didn't work. It just threw:

IOError: close() called during concurrent operation on the same file object.

and left the reader thread blocked on readline().
Bug https://bugzilla.mozilla.org/show_bug.cgi?id=1391545 which is recommended for disable is depending on this bug.

:whimboo, :gbrown do you guys have any updates regarding dis bug?
Flags: needinfo?(hskupin)
Flags: needinfo?(gbrown)
(Reporter)

Comment 4

6 days ago
I'm not working on this bug and don't have any new ideas for unblocking it.

This bug could theoretically cause bug 1391545, but I don't know if it *actually* is a cause of 1391545.
Flags: needinfo?(gbrown)
I don't have updates neither. A better approach would be to get Marionette client logging implemented via bug 1394381, which should also be on the deps list of bug 1391545. I will add it.
Flags: needinfo?(hskupin)
You need to log in before you can comment on or make changes to this bug.