1504308 - Error when running Python tests with ./mach test

Status: RESOLVED FIXED

(Testing :: Python Test, defect, P1)

Description

[Michael Cooper [:mythmon]]

When I run "./mach test <path/to/python/test.py>", I get the following output. This has been reproducible with multiple Python tests.


$ ./mach test toolkit/components/telemetry/tests/python/
Error running mach:

    ['test', 'toolkit/components/telemetry/tests/python/']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: coercing to Unicode: need string or buffer, NoneType found

  File "/home/mythmon/src/mozilla-unified/testing/mach_commands.py", line 169, in test
    argv=extra_args, test_objects=tests, **kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach/mach/registrar.py", line 129, in dispatch
    return self._run_command_handler(handler, context=context, **kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach/mach/registrar.py", line 90, in _run_command_handler
    result = fn(**kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach_commands.py", line 92, in python_test
    return self.run_python_tests(*args, **kwargs)
  File "/home/mythmon/src/mozilla-unified/python/mach_commands.py", line 105, in run_python_tests
    self.activate_pipenv(pipfile=None, populate=True, python=python)
  File "/home/mythmon/src/mozilla-unified/python/mozbuild/mozbuild/base.py", line 775, in activate_pipenv
    self.virtualenv_manager.activate_pipenv(pipfile, populate, python)
  File "/home/mythmon/src/mozilla-unified/python/mozbuild/mozbuild/virtualenv.py", line 621, in activate_pipenv
    stderr=subprocess.STDOUT, env=env)
  File "/usr/lib64/python2.7/subprocess.py", line 185, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 172, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib64/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception

Comment 1

[Andrew Halberstadt [:ahal]]

Not sure what's going on, but this is happening during `pipenv run python` when we populate the venv with the packages in virtualenv_packages.txt.

Comment 2

[Andrew Halberstadt [:ahal]]

Oh, I missed that this was running with |mach test|. Yeah I wouldn't expect that to work. Do you see the same issue with |mach python-test|?

The bug is still valid though as we should be supporting python tests in |mach test|.

Comment 3

[Michael Cooper [:mythmon]]

When using |mach python-test|, the test works as expected. It looks to me like |mach test| is calling |python-test| under the hood.

Comment 4

[Andrew Halberstadt [:ahal]]

Yeah, it's still a bug. All I meant was that I don't think anyone ever put in the effort to get |mach python-test| working properly with |mach test| so I'm not surprised that it's failing. I can reproduce this error too.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The command called in virtualenv.py is:

[u'/Users/henrik/code/gecko/obj/debug/_virtualenvs/init/bin/pipenv', u'run', u'python', u'/Users/henrik/code/gecko/python/mozbuild/mozbuild/virtualenv.py', u'populate', '/Users/henrik/code/gecko', u'/Users/henrik/code/gecko/obj/debug', None, u'/Users/henrik/code/gecko/build/virtualenv_packages.txt']

It means self.virtualenv_root is None, and that is the case because both the get_venv() and python are None. Mach simply misses to pass in the path to the Python interpreter.

It's easy to fix.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1504308 - [mach] Pass path to Python interpreter to python-tests. r=ahal

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/15f725097416
[mach] Pass path to Python interpreter to python-tests. r=ahal

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 15f725097416 (Bug 1504308) for python related build bustages

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=237671481&revision=15f725097416d14b300e407d11f3dcda726ca337

Backout link: https://hg.mozilla.org/integration/autoland/rev/71a1c61543e49af28e83e9d1b7f46e822cf4af9b

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=237671481&repo=autoland&lineNumber=41023

20:50:44 INFO - three=three, python=python, site_packages=site_packages
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\core.py", line 553, in ensure_virtualenv
20:50:44 INFO - if not project.virtualenv_exists:
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\project.py", line 225, in virtualenv_exists
20:50:44 INFO - if self.pipfile_exists and os.path.exists(self.virtualenv_location):
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\project.py", line 315, in virtualenv_location
20:50:44 INFO - loc = self._get_virtualenv_location(self.virtualenv_name)
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\project.py", line 240, in _get_virtualenv_location
20:50:44 INFO - if not venv.exists():
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\vendor\pathlib2.py", line 1451, in exists
20:50:44 INFO - self.stat()
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\vendor\pathlib2.py", line 1257, in stat
20:50:44 INFO - return self._accessor.stat(self)
20:50:44 INFO - File "z:\build\build\src\obj-firefox_virtualenvs\init\lib\site-packages\pipenv\vendor\pathlib2.py", line 481, in wrapped
20:50:44 INFO - return strfunc(str(pathobj), *args)
20:50:44 INFO - WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: 'z:\build\build\src\obj-firefox\_virtualenvs\obj-firefox-sNC2NmI7-z:\build\build\src\obj-firefox\_virtualenvs\init\Scripts\python.exe'
20:50:44 INFO - Error running mach:
20:50:44 INFO - ['--log-no-times', 'python-test', '-j16', '--subsuite', 'default']
20:50:44 INFO - The error occurred in code that was called by the mach command. This is either
20:50:44 INFO - a bug in the called code itself or in the way that mach is calling it.
20:50:44 INFO - You should consider filing a bug for this issue.
20:50:44 INFO - If filing a bug, please include the full output of mach, including this error
20:50:44 INFO - message.
20:50:44 INFO - The details of the failure are as follows:
20:50:44 INFO - CalledProcessError: Command '[u'z:/build/build/src/obj-firefox\_virtualenvs\init\Scripts\pipenv', u'--python', u'z:/build/build/src/obj-firefox\_virtualenvs\init\Scripts\python.exe']' returned non-zero exit status 1
20:50:44 INFO - File "z:/build/build/src\python/mach_commands.py", line 98, in python_test
20:50:44 INFO - return self.run_python_tests(*args, **kwargs)
20:50:44 INFO - File "z:/build/build/src\python/mach_commands.py", line 112, in run_python_tests
20:50:44 INFO - self.activate_pipenv(pipfile=None, populate=True, python=python)
20:50:44 INFO - File "z:/build/build/src\python/mozbuild\mozbuild\base.py", line 802, in activate_pipenv
20:50:44 INFO - self.virtualenv_manager.activate_pipenv(pipfile, populate, python)

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I'm a bit surprised what's failing here. Looks like a major bustage even for the build job. Andrew, could you please have a look at those failures? Do you have a hint for me? Thanks

Comment 11

[Andrew Halberstadt [:ahal]]

Some of the python tests (like the ones in python/mozbuild) get run as part of make check. So the build jobs do run mach python-test. For some reason it looks like the python path has two z:\build\build path segments in it. I have no idea why :/.

I guess you could go back to defaulting to 2.7 in your patch instead of the virtualenv one (but keep it where it is above the call to activate_pipenv). Though it would be worth fixing this if we can.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrew Halberstadt [:ahal] from comment #11)

Some of the python tests (like the ones in python/mozbuild) get run as part of make check. So the build jobs do run mach python-test. For some reason it looks like the python path has two z:\build\build path segments in it. I have no idea why :/.

I don't have the time to figure this out at the moment. I would suggest that we file a new bug to get rid of the default version for Python.

I guess you could go back to defaulting to 2.7 in your patch instead of the virtualenv one (but keep it where it is above the call to activate_pipenv). Though it would be worth fixing this if we can.

That I can indeed try. But it would only work if python would be not None so that self.virtualenv_manager.python_path is not getting used.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

New try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=69b58a9e42db8e

Comment 14

[Andrew Halberstadt [:ahal]]

Oh, I just meant instead of python = python or self.virtualenv_manager.python_path, you could do python = python or '2.7' (or set '2.7' as the default in the function signature).

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The above try worked fine. Do you think that we can land it that way? If yes, feel free to land it. I just uploaded the new version.

Comment 16

[Andrew Halberstadt [:ahal]]

Wfm.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0dd2c013c984
[mach] Pass path to Python interpreter to python-tests. r=ahal

Comment 18

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/0dd2c013c984