1766497 - virtualenv error with Python 3.10: No such file or directory: '[...]lib/python3.10/site-packages/mach.pth'

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect, P2)

Description

[Mathieu Leplatre [:leplatrem]]

I'm using MacOS Monterey 12.3.1 (apple chip), which ships Python 3.9.12 by default.

I installed Python 3.10 with brew and did brew link python@3.10

When running any ./mach command I get the following error:

created virtual environment CPython3.10.2.final.0-64 in 14ms
  creator CPython3Posix(dest=/Users/mathieu/.mozbuild/srcdirs/mozilla-unified-e2968b6c0d28/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Traceback (most recent call last):
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 96, in <module>
    main(sys.argv[1:])
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 88, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 35, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 21, in load_mach
    return mach_initialize.initialize(dir_path)
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/build/mach_initialize.py", line 153, in initialize
    _activate_python_environment(
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/build/mach_initialize.py", line 99, in _activate_python_environment
    mach_environment.activate()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 359, in activate
    self.ensure()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 346, in ensure
    self._build()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 384, in _build
    _create_venv_with_pthfile(
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 1189, in _create_venv_with_pthfile
    with open(os.path.join(platlib_site_packages_dir, PTH_FILENAME), "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/mathieu/.mozbuild/srcdirs/mozilla-unified-e2968b6c0d28/_virtualenvs/mach/lib/python3.10/site-packages/mach.pth'

The virtualenv folder does not have any site-packages folder. And mach does not seem to support it.

Switching to Python 3.9 works fine fortunately.

disclaimer: I'm a total newbie, with only a few days of experience on Mac OS

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Ah, I can reproduce this, thanks for the report!

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1766497: Mach: use `venv` instead of `virtualenv` r?ahal

Brew's Python 3.10 causes virtualenv==20.7.2 to produce a wonky folder
structure ($venv/opt/homebrew/lib/python3.10/site-packages?).

This is likely fixed with newer virtualenv, but the simpler workaround
here is to use venv instead now that Python 3 is always used.

Adds python3-venv to docker image so that tests and debian-based tasks
can leverage it.

Comment 3

[Mathieu Leplatre [:leplatrem]]

Just as a FYI, I upgraded to virtualenv==20.14.1 and face the same issue:

➜ virtualenv --version
virtualenv 20.14.1 from /opt/homebrew/lib/python3.10/site-packages/virtualenv/__init__.py

➜ ./mach lint
created virtual environment CPython3.10.2.final.0-64 in 14ms
  creator CPython3Posix(dest=/Users/mathieu/.mozbuild/srcdirs/mozilla-unified-e2968b6c0d28/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Traceback (most recent call last):
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 96, in <module>
    main(sys.argv[1:])
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 88, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 35, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 21, in load_mach
    return mach_initialize.initialize(dir_path)
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/build/mach_initialize.py", line 153, in initialize
    _activate_python_environment(
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/build/mach_initialize.py", line 99, in _activate_python_environment
    mach_environment.activate()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 359, in activate
    self.ensure()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 346, in ensure
    self._build()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 384, in _build
    _create_venv_with_pthfile(
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 1189, in _create_venv_with_pthfile
    with open(os.path.join(platlib_site_packages_dir, PTH_FILENAME), "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/mathieu/.mozbuild/srcdirs/mozilla-unified-e2968b6c0d28/_virtualenvs/mach/lib/python3.10/site-packages/mach.pth'

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

(In reply to Mathieu Leplatre [:leplatrem] from comment #3)

Just as a FYI, I upgraded to virtualenv==20.14.1 and face the same issue:

➜ virtualenv --version
virtualenv 20.14.1 from /opt/homebrew/lib/python3.10/site-packages/virtualenv/__init__.py

➜ ./mach lint
created virtual environment CPython3.10.2.final.0-64 in 14ms
  creator CPython3Posix(dest=/Users/mathieu/.mozbuild/srcdirs/mozilla-unified-e2968b6c0d28/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Traceback (most recent call last):
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 96, in <module>
    main(sys.argv[1:])
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 88, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 35, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/./mach", line 21, in load_mach
    return mach_initialize.initialize(dir_path)
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/build/mach_initialize.py", line 153, in initialize
    _activate_python_environment(
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/build/mach_initialize.py", line 99, in _activate_python_environment
    mach_environment.activate()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 359, in activate
    self.ensure()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 346, in ensure
    self._build()
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 384, in _build
    _create_venv_with_pthfile(
  File "/Users/mathieu/Code/Mozilla/firefox/mozilla-unified/python/mach/mach/site.py", line 1189, in _create_venv_with_pthfile
    with open(os.path.join(platlib_site_packages_dir, PTH_FILENAME), "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/mathieu/.mozbuild/srcdirs/mozilla-unified-e2968b6c0d28/_virtualenvs/mach/lib/python3.10/site-packages/mach.pth'

Firefox has its own vendored copy of virtualenv :)

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4b6378a06816
Mach: use `venv` instead of `virtualenv` r=ahal

Comment 6

[Norisz Fay [:noriszfay]]

Backed out for causing docker images bustages

Backout link

Push with failures

Failure log

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6369b96a0162
Mach: use `venv` instead of `virtualenv` . CLOSED TREE

Comment 8

[Narcis Beleuzu [:NarcisB]]

Backed out for python failures on test_site_compatibility.py

Backout link: https://hg.mozilla.org/integration/autoland/rev/cbaf5b946358bec9eefbcdafbfd12965593ec9f8
Log link: https://treeherder.mozilla.org/logviewer?job_id=376852912&repo=autoland&lineNumber=288

Please also check SM bustages -> https://treeherder.mozilla.org/logviewer?job_id=376852183&repo=autoland&lineNumber=641

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

I updated this a bit, but there's still an issue affecting Windows that I don't quite have time to get to the bottom of.

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1766497: Mozharness: use `venv` instead of `virtualenv` r?#build

Rather than using the version of pip (and associated tools) associated
with the current vendored virtualenv, instead directly use the one
vendored by Mach.

Note that, to give mozharness virtualenvs access to these libraries, a
"pthfile" is created - the logic to locate "site-packages" was copied
from "site.py".

Also, at this point, there's no need to use virtualenv when we can use
the simpler, faster, and standard-library venv module instead.

Depends on D144872

Comment 11

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1766497: Remove unused `virtualenv` package r?#build

Now that Python's stdlib venv is used, virtualenv is no longer
needed.

Depends on D145786

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

I've pushed two other optional WIP patches that (should?) allow us to drop virtualenv entirely. A couple things to note:

• I wasn't able to test these other patches on try due to the Ubuntu image issue from earlier. Here's a push I just submitted, unsure how green it'll be.
• Using venv is compatible with MozillaBuild 3.4 despite this issue, because we're using --without-pip, which avoids the problem.

Comment 13

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:ahochheiden, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 14

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1766497 - Updated a `log` call to use `command_context` instead of `self` (since `self` would never work in this instance) r?#build

Depends on D142274

Comment 15

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1766497 - Add `opener` parameter to `_MockBaseOpen` so that <PathObject>.open calls are mocked as expected r?#build

Depends on D148176

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/327fc5882333
Updated a `log` call to use `command_context` instead of `self` (since `self` would never work in this instance) r=firefox-build-system-reviewers,nalexander

Comment 17

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/327fc5882333

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/568656e2e997
Add `opener` parameter to `_MockBaseOpen` so that <PathObject>.open calls are mocked as expected r=ahal

Comment 19

[bszekely]

https://hg.mozilla.org/mozilla-central/rev/568656e2e997

Comment 22

[Greg Tatum [:gregtatum]]

I just ran into this after an unrelated brew install updated a bunch of dependencies, breaking my local environment. A work-around was to run:

brew unlink python@3.10
brew link python@3.9

Comment 23

[Greg Tatum [:gregtatum]]

I'm pretty sure this should be prioritized higher than a P3, since it breaks the common up to date mac configuration with a very obtuse error, unless I'm missing something.

Comment 24

[Alex Hochheiden [:ahochheiden] [he/him]]

(In reply to Greg Tatum [:gregtatum] from comment #23)

I'm pretty sure this should be prioritized higher than a P3, since it breaks the common up to date mac configuration with a very obtuse error, unless I'm missing something.

I'm fine with upgrading the priority. Initially this wasn't as much of a priority since it wasn't reported (and therefore assumed not to be encountered that often), implying not many people on macOS were using Python 3.10+. With the recent uptick in reports, it seems fair to make it a priority to resolve.

Comment 25

[Chris Peterson [:cpeterson]]

If the real fix is a lot of work, a temporary workaround might be for mach to print a more helpful error message (with this bug number and Homebrew instructions for using python@3.9) if it detects Python 3.10.

Comment 26

[BugBot [:suhaib / :marco]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit auto_nag documentation.

Comment 27

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1766497 - Upgrade virtualenv to version `20.16.3` r?#build

Comment 30

[Andrew McCreight [:mccr8]]

I've also hit this, FWIW.

Comment 31

[Jeff Muizelaar [:jrmuizel]]

I hit this as well.

Comment 32

[Nico Grunbaum [:ng, @chew:mozilla.org]]

I also hit this.

Comment 33

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I just ran into this with macOS 11.7 on Intel. Once I found this bug, I successfully worked around it with brew unlink python@3.10, and it now seems to be successfully using the system Python, which is 3.8.9. (I tried also unlinking/relinking python@3.9, but that doesn't create a python3 link in /usr/local/bin.) However, before I found this bug, I had no idea what to do.

Comment 34

[Gian-Carlo Pascutto [:gcp]]

If the suggested steps to fix don't work for you (on ARM macOS), make sure that any Intel homebrew python binaries are also unlinked. I was stuck here for a while until I realized that was the issue.

Comment 35

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1766497 - Doc updates regarding `venv` r?#build

• Removed references to Python 2
• Replaced references to virtualenv with venv

Depends on D145787

Comment 36

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1766497 - Remove early exit on Homebrew Python3.10 now that we're using `venv` instead of `virtualenv` r?#build

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/41be40696e00
Mach: use `venv` instead of `virtualenv` r=ahal
https://hg.mozilla.org/integration/autoland/rev/394555265c86
Mozharness: use `venv` instead of `virtualenv` r=ahal
https://hg.mozilla.org/integration/autoland/rev/aedf2d7af408
Remove unused `virtualenv` package r=ahal
https://hg.mozilla.org/integration/autoland/rev/6bb5f754957a
Doc updates regarding `venv` r=firefox-build-system-reviewers,nalexander
https://hg.mozilla.org/integration/autoland/rev/09f7f3d1c6dd
Remove early exit on Homebrew Python3.10 now that we're using `venv` instead of `virtualenv` r=firefox-build-system-reviewers,nalexander

Comment 38

[Cristian Tuns]

Backed out for causing build bustages with "No module named 'encodings' "

• Backout link
• Push with failures
• Failure Log
• Failure line: ModuleNotFoundError: No module named 'encodings'
  Return code: 3221226505
  3221226505 not in success codes: [0]
  Halting on failure while running ['z:\build\workspace\venv\Scripts\python', 'z:\build\build\src\third_party\python\_venv\wheels\pip-21.2.3-py3-none-any.whl\pip', 'install', '--only-binary', ':all:', '--disable-pip-version-check', 'z:\build\build\src\third_party\python\_venv\wheels\pip-21.2.3-py3-none-any.whl', 'z:\build\build\src\third_party\python\_venv\wheels\setuptools-51.2.0-py3-none-any.whl']

Comment 40

[Alex Hochheiden [:ahochheiden] [he/him]]

I wasn't able to resolve this after spending most of yesterday on it. I'm pretty sure it's a problem with the version of Python (3.6.5) being used by that job.

Luckily, that job is being upgraded to Python 3.9.10 alongside the Azure migration.

I ran the failing job with Azure, and it does not encounter the same error. Unless somebody else can think of a workaround to resolve the error on Python 3.6.5, I think this will be temporarily blocked until this job is using Azure to run.

I'm working with jmaher and mcornmesser to resolve the remaining Azure issues.

Comment 41

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1766497 - Manually copy `pyvenv.cfg` on Windows to workaround a bug with `venv` on Python 3.6 r?#build

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/30e0f3b255df
Mach: use `venv` instead of `virtualenv` r=ahal
https://hg.mozilla.org/integration/autoland/rev/ef004d353dd8
Mozharness: use `venv` instead of `virtualenv` r=ahal
https://hg.mozilla.org/integration/autoland/rev/9b776702e541
Remove unused `virtualenv` package r=ahal
https://hg.mozilla.org/integration/autoland/rev/339930d92fa1
Doc updates regarding `venv` r=firefox-build-system-reviewers,nalexander
https://hg.mozilla.org/integration/autoland/rev/a72a1a9be983
Remove early exit on Homebrew Python3.10 now that we're using `venv` instead of `virtualenv` r=firefox-build-system-reviewers,nalexander
https://hg.mozilla.org/integration/autoland/rev/923ed61551bb
Manually copy `pyvenv.cfg` on Windows to workaround a bug with `venv` on Python 3.6 r=firefox-build-system-reviewers,glandium

Comment 43

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/30e0f3b255df
https://hg.mozilla.org/mozilla-central/rev/ef004d353dd8
https://hg.mozilla.org/mozilla-central/rev/9b776702e541
https://hg.mozilla.org/mozilla-central/rev/339930d92fa1
https://hg.mozilla.org/mozilla-central/rev/a72a1a9be983
https://hg.mozilla.org/mozilla-central/rev/923ed61551bb