1657717 - Running python-test locally with python 3 twice throws FileNotFoundError

Status: RESOLVED FIXED

(Testing :: Python Test, defect)

Description

[Mitchell Hentges [:mhentges] 🦀]

To reproduce:

1. ./mach python-test --python 3 --subsuite mach (tests might fail)
2. ./mach python-test --python 3 --subsuite mach (tests won't run due to the following error:)

FileNotFoundError: [Errno 2] No such file or directory: '/home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/python'

  File "/home/mitch/dev/firefox/python/mach_commands.py", line 134, in python_test
    return self.run_python_tests(*args, **kwargs)
  File "/home/mitch/dev/firefox/python/mach_commands.py", line 149, in run_python_tests
    self._activate_test_virtualenvs(python)
  File "/home/mitch/dev/firefox/python/mach_commands.py", line 256, in _activate_test_virtualenvs
    self.activate_pipenv(pipfile=None, populate=True, python=python)
  File "/home/mitch/dev/firefox/python/mozbuild/mozbuild/base.py", line 859, in activate_pipenv
    self.ensure_pipenv()
  File "/home/mitch/dev/firefox/python/mozbuild/mozbuild/base.py", line 847, in ensure_pipenv
    self.activate_virtualenv()
  File "/home/mitch/dev/firefox/python/mozbuild/mozbuild/base.py", line 840, in activate_virtualenv
    self.virtualenv_manager.ensure()
  File "/home/mitch/dev/firefox/python/mozbuild/mozbuild/virtualenv.py", line 184, in ensure
    if self.up_to_date(python):
  File "/home/mitch/dev/firefox/python/mozbuild/mozbuild/virtualenv.py", line 153, in up_to_date
    if ((python, python_size) != (self.python_path, os.path.getsize(self.python_path)) and
  File "/usr/lib/python3.8/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Digging into this more, this is because:

1. I had pyenv installed, and Python 3.8.3 happened to have been installed
2. When I run ./mach python, the init_py3 venv that is created uses my global Python installation, which is the system installation: 3.8.2
3. When we create the pytest venv, we use pipenv. pipenv finds a newer version of python by searching pyenv, so creates the venv with 3.8.3 instead
4. Within the tests I was running, there was an invocation of mach that used the init_py3
5. When mach activates the init_py3 venv, it checks to make sure that it's up-to-date with the calling version of python (which it assumes to be the system python)
6. So, in combination with all of these factors, within the test invocation of mach:
   • The calling Python is 3.8.3 (from the test venv)
   • The existing init_py3 venv is using 3.8.2
   • This code finds that init_py3 is out-of-date, and deletes the python file
7. So, the next time that a mach command is run that needs the init_py3 venv, it fails, because the python binary is missing

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Note: an upcoming patch will resolve the behaviour of deleting a single binary, and will instead purge the entire venv so it can be safely re-created.

This resolves the error that was reported, but it causes thrashing on every mach python-test run that calls mach internally:

• When mach python-test runs, it wipes init_py3 and sets it up with 3.8.2
• Then, during that^ process, when the nested mach call happens, the init_py3 is wiped again and replaced with 3.8.3

Based on the knowledge I have so far here, I'm guessing that the solution will be to always create our virtualenvs with the same python version (the system version). So, we'll probably just need to bend pipenv to only use the system python, and to not check pyenv.

Comment 3

[Ricky Stewart]

So, we'll probably just need to bend pipenv to only use the system python, and to not check pyenv.

Backwards: we want to bend pipenv to only use the Python that we're running python-test with, rather than doing its own custom search and trying to guess we mean a different version. The system Python is not necessarily correct, and in fact the system Python will NEVER be the correct Python after https://phabricator.services.mozilla.com/D85916.

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1657717: pipenv should use same version of python as currently active, if possible

Pipenv enthusiastically checks pyenv for possible Python installations, and it seems to always want to use the most
up-to-date version possible. This can lead to issues if the version of python used for the regular venvs is older
than another version of python that exists on the machine (such as if the system python is 3.8.2, but pyenv has
3.8.3 installed).

This patch addresses this by scoping pipenv to match the currently-running version of python (if major versions line
up).

Depends on D86269

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d141ba87d0c1
pipenv should use same version of python as currently active, if possible r=firefox-build-system-reviewers,rstewart

Comment 7

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/d141ba87d0c1