source-test-python-mozbase persistently fails on macosx1014 due to ModuleNotFoundError: No module named 'six'
Categories
(Testing :: Mozbase, defect)
Tracking
(firefox-esr68 fixed, firefox69 fixed)
People
(Reporter: egao, Assigned: egao)
References
Details
Attachments
(1 file)
|
47 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr68+
|
Details | Review |
Summary:
Four tests run as part of the mozbase subsuite in source-test-python-mozbase-macosx1014-64/opt-py3 py3(mb) persistently fails on the new macosx1014-mojave environment:
0:47.84 testing/mozbase/mozprocess/test/test_misc.py::ProcTestMisc::test_process_timeout_no_kill TEST-UNEXPECTED-FAIL
0:50.80 testing/mozbase/mozprocess/tests/test_wait.py::ProcTestWait::test_timeout TEST-UNEXPECTED-FAIL
0:55.75 testing/mozbase/mozprocess/test/test_kill.py::ProcTestKill::test_process_kill_broad_delayed TEST-UNEXPECTED-FAIL
0:55.75 testing/mozbase/mozprocess/test/test_kill.py::ProcTestKill::test_process_kill_deep_wait TEST-UNEXPECTED-FAIL```
Cause
Pytest outputs the cause of failure:
[task 2019-06-21T16:46:27.723Z] 0:47.84 testing/mozbase/mozprocess/tests/test_misc.py::ProcTestMisc::test_process_timeout_no_kill TEST-UNEXPECTED-FAIL
[task 2019-06-21T16:46:27.723Z] 0:47.84 testing/mozbase/mozprocess/tests/test_misc.py::ProcTestMisc::test_unicode_in_environment PASSED
[task 2019-06-21T16:46:27.723Z] 0:47.84
[task 2019-06-21T16:46:27.723Z] 0:47.84 =================================== FAILURES ===================================
[task 2019-06-21T16:46:27.723Z] 0:47.84 __________________ ProcTestMisc.test_process_timeout_no_kill ___________________
[task 2019-06-21T16:46:27.724Z] 0:47.84
[task 2019-06-21T16:46:27.724Z] 0:47.84 self = <test_misc.ProcTestMisc testMethod=test_process_timeout_no_kill>
[task 2019-06-21T16:46:27.724Z] 0:47.85
[task 2019-06-21T16:46:27.724Z] 0:47.85 def test_process_timeout_no_kill(self):
[task 2019-06-21T16:46:27.724Z] 0:47.85 """ Process is started, runs but we time out waiting on it
[task 2019-06-21T16:46:27.724Z] 0:47.85 to complete. Process should not be killed.
[task 2019-06-21T16:46:27.724Z] 0:47.85 """
[task 2019-06-21T16:46:27.724Z] 0:47.85 p = None
[task 2019-06-21T16:46:27.724Z] 0:47.85
[task 2019-06-21T16:46:27.724Z] 0:47.85 def timeout_handler():
[task 2019-06-21T16:46:27.724Z] 0:47.85 self.assertEqual(p.proc.poll(), None)
[task 2019-06-21T16:46:27.725Z] 0:47.85 p.kill()
[task 2019-06-21T16:46:27.725Z] 0:47.85 p = processhandler.ProcessHandler([self.python, self.proclaunch,
[task 2019-06-21T16:46:27.725Z] 0:47.85 "process_waittimeout.ini"],
[task 2019-06-21T16:46:27.725Z] 0:47.85 cwd=here,
[task 2019-06-21T16:46:27.725Z] 0:47.85 onTimeout=(timeout_handler,),
[task 2019-06-21T16:46:27.725Z] 0:47.85 kill_on_timeout=False)
[task 2019-06-21T16:46:27.725Z] 0:47.85 p.run(timeout=1)
[task 2019-06-21T16:46:27.725Z] 0:47.85 p.wait()
[task 2019-06-21T16:46:27.725Z] 0:47.85 > self.assertTrue(p.didTimeout)
[task 2019-06-21T16:46:27.725Z] 0:47.85 E AssertionError: False is not true
[task 2019-06-21T16:46:27.726Z] 0:47.85
[task 2019-06-21T16:46:27.726Z] 0:47.85 testing/mozbase/mozprocess/tests/test_misc.py:35: AssertionError
[task 2019-06-21T16:46:27.726Z] 0:47.85 ----------------------------- Captured stdout call -----------------------------
[task 2019-06-21T16:46:27.726Z] 0:47.85 Traceback (most recent call last):
[task 2019-06-21T16:46:27.726Z] 0:47.85 File "/Users/cltbld/tasks/task_1561130078/checkouts/gecko/testing/mozbase/mozprocess/tests/proclaunch.py", line 10, in <module>
[task 2019-06-21T16:46:27.726Z] 0:47.85 from six.moves import configparser
[task 2019-06-21T16:46:27.726Z] 0:47.85 ModuleNotFoundError: No module named 'six'
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 1•5 years ago
|
||
Using this log as example: https://taskcluster-artifacts.net/XTXNxf4tT6WPNS8V8DfQIg/0/public/logs/live_backing.log
When a task is started for mozbase, the following takes place:
run-taskcalls the test specifying a python version and the subsuite asmozbase(eg./usr/local/bin/python3 run-task '--gecko-checkout=./checkouts/gecko' -- bash -cx 'cd $GECKO_PATH && LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 ./mach python-test --python /usr/local/bin/python3 --subsuite mozbase')- initial virutalenv is created using python2.7
- psutil is built and installed to python2.7 virtual environment (uninteresting to us)
- using a hardcoded list (code), vendored dependecies are installed to the python2.7 virtual environment
- pipenv kicks in and initializes a virtualenv using the version of python specified in the command line (code)
- pipenv (attempts) to populate dependencies from manifest file
This is where things become somewhat confusing and unclear.
According to this portion of virtualenv.py, the manifest file should be parsed and used to populate multiple packages, including third_party/python/six. What this populate does is uncertain.
|
||
Comment 2•5 years ago
|
||
I don't believe this has anything to do with the mach/mozilla-central packaging. It's a specific test that is running python in a subprocess:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/tests/test_misc.py#28
I think we can solve it by passing in the things that are in sys.path to the subprocess via the PYTHONPATH environment variable. Or at least pass in six.
|
|
Assignee | |
Comment 3•5 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #2)
I don't believe this has anything to do with the mach/mozilla-central packaging. It's a specific test that is running python in a subprocess:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/tests/test_misc.py#28I think we can solve it by passing in the things that are in
sys.pathto the subprocess via thePYTHONPATHenvironment variable. Or at least pass insix.
Some time ago I did attempt to overwrite, or at least add to the PYTHONPATH/PATH environment variable the location of six. This was attempted using two methods, os.environ and sys.path but neither of them seems to have worked.
With that said, I'll attempt it again, perhaps I had the right approach but in the wrong section of the code.
|
|
||
Comment 4•5 years ago
|
||
You'll need to create a dict that contains PYTHONPATH and pass it into the env parameter of the ProcessHandler call.
|
|
Assignee | |
Comment 5•5 years ago
|
||
|
|
Assignee | |
Updated•5 years ago
|
Pushed by egao@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/da3af6a077fc explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests r=jmaher
|
||
Comment 7•5 years ago
|
||
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=254619239&repo=autoland&lineNumber=499
Backout link: https://hg.mozilla.org/integration/autoland/rev/be232271fcde16cffc84ef2f24a8e849302c5991
|
|
Assignee | |
Comment 8•5 years ago
|
||
Sorry - it was a simple mistake that should have been caught by me prior to landing. I have a new try push with the fixed patch:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8dd28594ad9eb2830c7e9069e866c8d4aa1bcafe
|
|
Assignee | |
Comment 9•5 years ago
|
||
New patch passes Decision Task, runs tests fine.
|
||
Comment 10•5 years ago
|
||
Pushed by egao@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/6ed76105f2b3 explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests r=jmaher
|
||
Comment 11•5 years ago
|
||
Comment on attachment 9075523 [details]
Bug 1562083 - explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests
ESR Uplift Approval Request
- If this is not a sec:{high,crit} bug, please state case for ESR consideration: support for running tests on mac osx 10.14 in order to reimage our static pool away from osx 10.10.
- User impact if declined:
- Fix Landed on Version: 69
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky):
- String or UUID changes made by this patch:
|
||
Comment 12•5 years ago
|
||
| bugherder | ||
|
||
Comment 13•5 years ago
|
||
Comment on attachment 9075523 [details]
Bug 1562083 - explicitly insert sys.path to env for use in subprocess for python3 mozprocess unit tests
Test-only fix needed to run tests on OSX 10.14. Approved for 68.1esr.
|
|
||
Comment 14•5 years ago
|
||
| bugherder uplift | ||
Description
•