1325509 - Marionette tests can no longer be run from |multiprocessing.Pool| tasks

Status: RESOLVED WONTFIX

(Remote Protocol :: Marionette, defect, P3)

Description

[Eric Rahm [:erahm]]

AWSY [1] uses a |multiprocessing.Pool| to manage concurrent memory tests [2]. After upgrading from marionette-client==3.1.0 -> marionette-harness==4.0.0 I started seeing the following exception:

> daemonic processes are not allowed to have children

It appears that we've changed how the Firefox process is launched such that it's upsetting |multiprocessing.Pool|. The common recommendation for fixing such an issue is to switch over to a |multiprocessing.pool.ThreadPool| [3], which does indeed fix the exception but leads to test failures (I think Marionette explicitly disallows running multiple tests at once).

I will follow up with a reduced test case.

[1] https://github.com/mozilla/areweslimyet
[2] https://github.com/mozilla/areweslimyet/blob/0b23e6a7a88a079780b3db961cd7d92f064cc9b0/benchtester/BatchTester.py#L260
[3] http://stackoverflow.com/a/34069003

Comment 1

[Eric Rahm [:erahm]]

:whimboo, Maja suggested this may have been introduced via bug 1283906.

Comment 2

[Eric Rahm [:erahm]]

Attachment: pool_bug.tar.bz2

This is a reduced example (sorry for the multiple files, that's just how marionette works).

Usage:
  1. Edit pool_bug.py, set |BINARY| to your firefox binary
  2. |./run_test.sh|

Note that 4 browsers are opened, but the test harness bails with the aforementioned exception.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Eric, it would be great if you could add the exception you are seeing as comment. It might help us to better find out what's wrong.

I have to add that I didn't change anything how Firefox gets launched via any of the dependent bugs on the meta bug 1283906. I assume marionette-client==3.2.0 and marionette-client==3.3.0 also fail for you? For a change log of those releases see bug 1309899 and bug 1320073.

Comment 4

[Eric Rahm [:erahm]]

(In reply to Henrik Skupin (:whimboo) from comment #3)
> Eric, it would be great if you could add the exception you are seeing as
> comment. It might help us to better find out what's wrong.
> 
> I have to add that I didn't change anything how Firefox gets launched via
> any of the dependent bugs on the meta bug 1283906. I assume
> marionette-client==3.2.0 and marionette-client==3.3.0 also fail for you? For
> a change log of those releases see bug 1309899 and bug 1320073.

You cant test with the reduced test case that is attached.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sorry, I don't have the time for that right now. If you can do it, it would be of big help. Once we know that a patch from myself caused it, I can make sure to get it fixed. Thanks.

Comment 6

[Eric Rahm [:erahm]]

> Traceback (most recent call last):
>   File "pool_bug.py", line 28, in run_test
>     runner.run_tests("test_pool_bug.py")
>   File "/home/erahm/dev/tmp/pool_bug/test_env/local/lib/python2.7/site-packages/marionette_harness/runner/base.py", line 815, in run_tests
>     self.fixture_servers = self.start_fixture_servers()
>   File "/home/erahm/dev/tmp/pool_bug/test_env/local/lib/python2.7/site-packages/marionette_harness/runner/base.py", line 908, in start_fixture_servers
>     return serve.start(root)
>   File "/home/erahm/dev/tmp/pool_bug/test_env/local/lib/python2.7/site-packages/marionette_harness/runner/serve.py", line 176, in start
>     servers = start_servers(doc_root, ssl_config, **kwargs)
>   File "/home/erahm/dev/tmp/pool_bug/test_env/local/lib/python2.7/site-packages/marionette_harness/runner/serve.py", line 156, in start_servers
>     proc.start(doc_root, ssl_config, **kwargs)
>   File "/home/erahm/dev/tmp/pool_bug/test_env/local/lib/python2.7/site-packages/marionette_harness/runner/serve.py", line 112, in start
>     self.proc.start()
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 124, in start
>     'daemonic processes are not allowed to have children'
> AssertionError: daemonic processes are not allowed to have children

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so that's a regression from bug 1103196 then. Andreas is the one you are looking for.

Comment 8

[Andreas Tolfsen ❲:ato❳]

A wild guess is that it is because we’re setting multiprocessing.Process.daemon to True here: https://github.com/mozilla/gecko-dev/blob/master/testing/marionette/harness/marionette_harness/runner/serve.py#L111

You should be able to confirm this theory by removing that line and rerunning your existing tests.

Comment 9

[Eric Rahm [:erahm]]

(In reply to Andreas Tolfsen ‹:ato› from comment #8)
> A wild guess is that it is because we’re setting
> multiprocessing.Process.daemon to True here:
> https://github.com/mozilla/gecko-dev/blob/master/testing/marionette/harness/
> marionette_harness/runner/serve.py#L111
> 
> You should be able to confirm this theory by removing that line and
> rerunning your existing tests.

Testing with |daemon = False| still results in failure. I believe the assertion we're hitting is that the *parent* process cannot be daemonic [1]. Since I'm using a pool the parent is daemonic.

I've provided a reduced test case in attachment 8821396 [details] to make it easier for you to test locally.

[1] https://github.com/python/cpython/blob/2.7/Lib/multiprocessing/process.py#L123-L124

Comment 10

[Eric Rahm [:erahm]]

AWSY no longer uses multiprocessing, lets just close this.