Closed Bug 1562083 Opened 4 months ago Closed 4 months ago

source-test-python-mozbase persistently fails on macosx1014 due to ModuleNotFoundError: No module named 'six'

Categories

(Testing :: Mozbase, defect)

defect
Not set

Tracking

(firefox-esr68 fixed, firefox69 fixed)

RESOLVED FIXED
mozilla69
Tracking Status
firefox-esr68 --- fixed
firefox69 --- fixed

People

(Reporter: egao, Assigned: egao)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Summary:

Four tests run as part of the mozbase subsuite in source-test-python-mozbase-macosx1014-64/opt-py3 py3(mb) persistently fails on the new macosx1014-mojave environment:

0:47.84 testing/mozbase/mozprocess/test/test_misc.py::ProcTestMisc::test_process_timeout_no_kill TEST-UNEXPECTED-FAIL
0:50.80 testing/mozbase/mozprocess/tests/test_wait.py::ProcTestWait::test_timeout TEST-UNEXPECTED-FAIL
0:55.75 testing/mozbase/mozprocess/test/test_kill.py::ProcTestKill::test_process_kill_broad_delayed TEST-UNEXPECTED-FAIL
0:55.75 testing/mozbase/mozprocess/test/test_kill.py::ProcTestKill::test_process_kill_deep_wait TEST-UNEXPECTED-FAIL```

Cause

Pytest outputs the cause of failure:

[task 2019-06-21T16:46:27.723Z]  0:47.84 testing/mozbase/mozprocess/tests/test_misc.py::ProcTestMisc::test_process_timeout_no_kill TEST-UNEXPECTED-FAIL
[task 2019-06-21T16:46:27.723Z]  0:47.84 testing/mozbase/mozprocess/tests/test_misc.py::ProcTestMisc::test_unicode_in_environment PASSED
[task 2019-06-21T16:46:27.723Z]  0:47.84 
[task 2019-06-21T16:46:27.723Z]  0:47.84 =================================== FAILURES ===================================
[task 2019-06-21T16:46:27.723Z]  0:47.84 __________________ ProcTestMisc.test_process_timeout_no_kill ___________________
[task 2019-06-21T16:46:27.724Z]  0:47.84 
[task 2019-06-21T16:46:27.724Z]  0:47.84 self = <test_misc.ProcTestMisc testMethod=test_process_timeout_no_kill>
[task 2019-06-21T16:46:27.724Z]  0:47.85 
[task 2019-06-21T16:46:27.724Z]  0:47.85     def test_process_timeout_no_kill(self):
[task 2019-06-21T16:46:27.724Z]  0:47.85         """ Process is started, runs but we time out waiting on it
[task 2019-06-21T16:46:27.724Z]  0:47.85                 to complete. Process should not be killed.
[task 2019-06-21T16:46:27.724Z]  0:47.85             """
[task 2019-06-21T16:46:27.724Z]  0:47.85         p = None
[task 2019-06-21T16:46:27.724Z]  0:47.85 
[task 2019-06-21T16:46:27.724Z]  0:47.85         def timeout_handler():
[task 2019-06-21T16:46:27.724Z]  0:47.85             self.assertEqual(p.proc.poll(), None)
[task 2019-06-21T16:46:27.725Z]  0:47.85             p.kill()
[task 2019-06-21T16:46:27.725Z]  0:47.85         p = processhandler.ProcessHandler([self.python, self.proclaunch,
[task 2019-06-21T16:46:27.725Z]  0:47.85                                            "process_waittimeout.ini"],
[task 2019-06-21T16:46:27.725Z]  0:47.85                                           cwd=here,
[task 2019-06-21T16:46:27.725Z]  0:47.85                                           onTimeout=(timeout_handler,),
[task 2019-06-21T16:46:27.725Z]  0:47.85                                           kill_on_timeout=False)
[task 2019-06-21T16:46:27.725Z]  0:47.85         p.run(timeout=1)
[task 2019-06-21T16:46:27.725Z]  0:47.85         p.wait()
[task 2019-06-21T16:46:27.725Z]  0:47.85 >       self.assertTrue(p.didTimeout)
[task 2019-06-21T16:46:27.725Z]  0:47.85 E       AssertionError: False is not true
[task 2019-06-21T16:46:27.726Z]  0:47.85 
[task 2019-06-21T16:46:27.726Z]  0:47.85 testing/mozbase/mozprocess/tests/test_misc.py:35: AssertionError
[task 2019-06-21T16:46:27.726Z]  0:47.85 ----------------------------- Captured stdout call -----------------------------
[task 2019-06-21T16:46:27.726Z]  0:47.85 Traceback (most recent call last):
[task 2019-06-21T16:46:27.726Z]  0:47.85   File "/Users/cltbld/tasks/task_1561130078/checkouts/gecko/testing/mozbase/mozprocess/tests/proclaunch.py", line 10, in <module>
[task 2019-06-21T16:46:27.726Z]  0:47.85     from six.moves import configparser
[task 2019-06-21T16:46:27.726Z]  0:47.85 ModuleNotFoundError: No module named 'six'
Component: RelOps: Posix OS → Mozbase
Product: Infrastructure & Operations → Testing

Using this log as example: https://taskcluster-artifacts.net/XTXNxf4tT6WPNS8V8DfQIg/0/public/logs/live_backing.log

When a task is started for mozbase, the following takes place:

  1. run-task calls the test specifying a python version and the subsuite as mozbase (eg. /usr/local/bin/python3 run-task '--gecko-checkout=./checkouts/gecko' -- bash -cx 'cd $GECKO_PATH && LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 ./mach python-test --python /usr/local/bin/python3 --subsuite mozbase')
  2. initial virutalenv is created using python2.7
  3. psutil is built and installed to python2.7 virtual environment (uninteresting to us)
  4. using a hardcoded list (code), vendored dependecies are installed to the python2.7 virtual environment
  5. pipenv kicks in and initializes a virtualenv using the version of python specified in the command line (code)
  6. pipenv (attempts) to populate dependencies from manifest file

This is where things become somewhat confusing and unclear.

According to this portion of virtualenv.py, the manifest file should be parsed and used to populate multiple packages, including third_party/python/six. What this populate does is uncertain.

I don't believe this has anything to do with the mach/mozilla-central packaging. It's a specific test that is running python in a subprocess:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/tests/test_misc.py#28

I think we can solve it by passing in the things that are in sys.path to the subprocess via the PYTHONPATH environment variable. Or at least pass in six.

(In reply to Andrew Halberstadt [:ahal] from comment #2)

I don't believe this has anything to do with the mach/mozilla-central packaging. It's a specific test that is running python in a subprocess:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/tests/test_misc.py#28

I think we can solve it by passing in the things that are in sys.path to the subprocess via the PYTHONPATH environment variable. Or at least pass in six.

Some time ago I did attempt to overwrite, or at least add to the PYTHONPATH/PATH environment variable the location of six. This was attempted using two methods, os.environ and sys.path but neither of them seems to have worked.

With that said, I'll attempt it again, perhaps I had the right approach but in the wrong section of the code.

You'll need to create a dict that contains PYTHONPATH and pass it into the env parameter of the ProcessHandler call.

See Also: → 1551603
Assignee: nobody → egao
Status: NEW → ASSIGNED
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/da3af6a077fc
explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests r=jmaher

Sorry - it was a simple mistake that should have been caught by me prior to landing. I have a new try push with the fixed patch:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8dd28594ad9eb2830c7e9069e866c8d4aa1bcafe

Flags: needinfo?(egao)

New patch passes Decision Task, runs tests fine.

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6ed76105f2b3
explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests r=jmaher

Comment on attachment 9075523 [details]
Bug 1562083 - explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: support for running tests on mac osx 10.14 in order to reimage our static pool away from osx 10.10.
  • User impact if declined:
  • Fix Landed on Version: 69
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky):
  • String or UUID changes made by this patch:
Attachment #9075523 - Flags: approval-mozilla-esr68?
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla69

Comment on attachment 9075523 [details]
Bug 1562083 - explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests

Test-only fix needed to run tests on OSX 10.14. Approved for 68.1esr.

Attachment #9075523 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
You need to log in before you can comment on or make changes to this bug.