Error when running Python tests with ./mach test

RESOLVED FIXED in Firefox 68

Status

defect
P1
normal
RESOLVED FIXED
9 months ago
4 months ago

People

(Reporter: mythmon, Assigned: whimboo)

Tracking

(Blocks 1 bug)

Version 3
mozilla68
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(1 attachment)

When I run "./mach test <path/to/python/test.py>", I get the following output. This has been reproducible with multiple Python tests.


$ ./mach test toolkit/components/telemetry/tests/python/
Error running mach:

    ['test', 'toolkit/components/telemetry/tests/python/']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: coercing to Unicode: need string or buffer, NoneType found

  File "/home/mythmon/src/mozilla-unified/testing/mach_commands.py", line 169, in test
    argv=extra_args, test_objects=tests, **kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach/mach/registrar.py", line 129, in dispatch
    return self._run_command_handler(handler, context=context, **kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach/mach/registrar.py", line 90, in _run_command_handler
    result = fn(**kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach_commands.py", line 92, in python_test
    return self.run_python_tests(*args, **kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach_commands.py", line 105, in run_python_tests
    self.activate_pipenv(pipfile=None, populate=True, python=python)
  File "/home/mythmon/src/mozilla-unified/python/mozbuild/mozbuild/base.py", line 775, in activate_pipenv
    self.virtualenv_manager.activate_pipenv(pipfile, populate, python)
  File "/home/mythmon/src/mozilla-unified/python/mozbuild/mozbuild/virtualenv.py", line 621, in activate_pipenv
    stderr=subprocess.STDOUT, env=env)
  File "/usr/lib64/python2.7/subprocess.py", line 185, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 172, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib64/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
Not sure what's going on, but this is happening during `pipenv run python` when we populate the venv with the packages in virtualenv_packages.txt.
Oh, I missed that this was running with |mach test|. Yeah I wouldn't expect that to work. Do you see the same issue with |mach python-test|?

The bug is still valid though as we should be supporting python tests in |mach test|.
When using |mach python-test|, the test works as expected. It looks to me like |mach test| is calling |python-test| under the hood.
Yeah, it's still a bug. All I meant was that I don't think anyone ever put in the effort to get |mach python-test| working properly with |mach test| so I'm not surprised that it's failing. I can reproduce this error too.
Priority: -- → P3
Duplicate of this bug: 1518200

The command called in virtualenv.py is:

[u'/Users/henrik/code/gecko/obj/debug/_virtualenvs/init/bin/pipenv', u'run', u'python', u'/Users/henrik/code/gecko/python/mozbuild/mozbuild/virtualenv.py', u'populate', '/Users/henrik/code/gecko', u'/Users/henrik/code/gecko/obj/debug', None, u'/Users/henrik/code/gecko/build/virtualenv_packages.txt']

It means self.virtualenv_root is None, and that is the case because both the get_venv() and python are None. Mach simply misses to pass in the path to the Python interpreter.

It's easy to fix.

Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Priority: P3 → P1
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/15f725097416
[mach] Pass path to Python interpreter to python-tests. r=ahal

Backed out changeset 15f725097416 (Bug 1504308) for python related build bustages

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=237671481&revision=15f725097416d14b300e407d11f3dcda726ca337

Backout link: https://hg.mozilla.org/integration/autoland/rev/71a1c61543e49af28e83e9d1b7f46e822cf4af9b

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237671481&repo=autoland&lineNumber=41023

20:50:44 INFO - three=three, python=python, site_packages=site_packages
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\core.py", line 553, in ensure_virtualenv
20:50:44 INFO - if not project.virtualenv_exists:
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\project.py", line 225, in virtualenv_exists
20:50:44 INFO - if self.pipfile_exists and os.path.exists(self.virtualenv_location):
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\project.py", line 315, in virtualenv_location
20:50:44 INFO - loc = self._get_virtualenv_location(self.virtualenv_name)
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\project.py", line 240, in _get_virtualenv_location
20:50:44 INFO - if not venv.exists():
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\vendor\pathlib2.py", line 1451, in exists
20:50:44 INFO - self.stat()
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\vendor\pathlib2.py", line 1257, in stat
20:50:44 INFO - return self._accessor.stat(self)
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\vendor\pathlib2.py", line 481, in wrapped
20:50:44 INFO - return strfunc(str(pathobj), *args)
20:50:44 INFO - WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: 'z:\build\build\src\obj-firefox\_virtualenvs\obj-firefox-sNC2NmI7-z:\build\build\src\obj-firefox\_virtualenvs\init\Scripts\python.exe'
20:50:44 INFO - Error running mach:
20:50:44 INFO - ['--log-no-times', 'python-test', '-j16', '--subsuite', 'default']
20:50:44 INFO - The error occurred in code that was called by the mach command. This is either
20:50:44 INFO - a bug in the called code itself or in the way that mach is calling it.
20:50:44 INFO - You should consider filing a bug for this issue.
20:50:44 INFO - If filing a bug, please include the full output of mach, including this error
20:50:44 INFO - message.
20:50:44 INFO - The details of the failure are as follows:
20:50:44 INFO - CalledProcessError: Command '[u'z:/build/build/src/obj-firefox\_virtualenvs\init\Scripts\pipenv', u'--python', u'z:/build/build/src/obj-firefox\_virtualenvs\init\Scripts\python.exe']' returned non-zero exit status 1
20:50:44 INFO - File "z:/build/build/src\python/mach_commands.py", line 98, in python_test
20:50:44 INFO - return self.run_python_tests(*args, **kwargs)
20:50:44 INFO - File "z:/build/build/src\python/mach_commands.py", line 112, in run_python_tests
20:50:44 INFO - self.activate_pipenv(pipfile=None, populate=True, python=python)
20:50:44 INFO - File "z:/build/build/src\python/mozbuild\mozbuild\base.py", line 802, in activate_pipenv
20:50:44 INFO - self.virtualenv_manager.activate_pipenv(pipfile, populate, python)

Flags: needinfo?(hskupin)

I'm a bit surprised what's failing here. Looks like a major bustage even for the build job. Andrew, could you please have a look at those failures? Do you have a hint for me? Thanks

Flags: needinfo?(hskupin) → needinfo?(ahal)

Some of the python tests (like the ones in python/mozbuild) get run as part of make check. So the build jobs do run mach python-test. For some reason it looks like the python path has two z:\build\build path segments in it. I have no idea why :/.

I guess you could go back to defaulting to 2.7 in your patch instead of the virtualenv one (but keep it where it is above the call to activate_pipenv). Though it would be worth fixing this if we can.

Flags: needinfo?(ahal)

(In reply to Andrew Halberstadt [:ahal] from comment #11)

Some of the python tests (like the ones in python/mozbuild) get run as part of make check. So the build jobs do run mach python-test. For some reason it looks like the python path has two z:\build\build path segments in it. I have no idea why :/.

I don't have the time to figure this out at the moment. I would suggest that we file a new bug to get rid of the default version for Python.

I guess you could go back to defaulting to 2.7 in your patch instead of the virtualenv one (but keep it where it is above the call to activate_pipenv). Though it would be worth fixing this if we can.

That I can indeed try. But it would only work if python would be not None so that self.virtualenv_manager.python_path is not getting used.

Oh, I just meant instead of python = python or self.virtualenv_manager.python_path, you could do python = python or '2.7' (or set '2.7' as the default in the function signature).

The above try worked fine. Do you think that we can land it that way? If yes, feel free to land it. I just uploaded the new version.

Flags: needinfo?(ahal)

Wfm.

Flags: needinfo?(ahal)
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0dd2c013c984
[mach] Pass path to Python interpreter to python-tests. r=ahal
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
You need to log in before you can comment on or make changes to this bug.