1670039 - |mach create-mach-environment| failed to import "virtualenv.seed.via_app_data" on Arch Linux

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect, P3)

Description

[Mitchell Hentges [:mhentges] 🦀]

Discussion on Matrix.

Full log:

Bootstrapping...
Using an experimental bootstrapper for Archlinux.

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
Your choice: 2
Your version of Python 3 (3.8.6) is new enough.
Your version of Python 2 (2.7.18) is new enough.
Traceback (most recent call last):
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 46, in <module>
    run()
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv.py", line 42, in run
    run_virtualenv()
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/__main__.py", line 16, in run
    session = cli_run(args, options)
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/run/__init__.py", line 26, in cli_run
    of_session = session_via_cli(args, options, setup_logging)
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/run/__init__.py", line 42, in session_via_cli
    parser, elements = build_parser(args, options, setup_logging)
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/run/__init__.py", line 69, in build_parser
    SeederSelector(interpreter, parser),
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/run/plugin/seeders.py", line 8, in __init__
    possible = self.options("virtualenv.seed")
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 39, in options
    cls._OPTIONS = cls.entry_points_for(key)
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 18, in entry_points_for
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 18, in <genexpr>
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
Error running mach:

    ['create-mach-environment']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file create-mach-environment| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Failed to create virtualenv: /var/tmp/makepkg/firefox-nightly-hg/src/build-profile-generate/_virtualenvs/mach (virtualenv.py retcode: 1)

  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 1547, in create_mach_environment
    manager.build(sys.executable)
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/python/mozbuild/mozbuild/virtualenv.py", line 490, in build
    self.create(python)
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/python/mozbuild/mozbuild/virtualenv.py", line 264, in create
    raise Exception(
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/usr/bin/python3', '/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/mach', 'create-mach-environment']' returned non-zero exit status 1.

  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 45, in bootstrap
    bootstrapper.bootstrap()
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 368, in bootstrap
    self.instance.ensure_mach_environment(checkout_root)
  File "/var/tmp/makepkg/firefox-nightly-hg/src/mozilla-central/python/mozboot/mozboot/base.py", line 277, in ensure_mach_environment
    subprocess.check_call(cmd, cwd=checkout_root)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

FYI, I'm seeing this ./mach create-mach-environment on Ubuntu.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

That's great news, I'd like to avoid setting up an Arch install if possible :)
What is your python3 --version? Which version of Ubuntu are you using (20.04?)
And, if you destroy your virtualenvs (rm -rf ~/.mozbuild/_virtualenvs) and run ./mach create-mach-environment, can you still reproduce?
Finally, if you can still repro, can you submit your full ./mach create-mach-environment log here?

I'm off to lunch here, but I'll follow-up after. Thanks :)

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Ubuntu 20.04.1 LTS

$  python3 --version
Python 3.9.0
$  pyenv global
3.9.0
2.7.18

Still broken after destroying virtualenvs:

 ./mach create-mach-environment
created virtual environment CPython3.9.0.final.0-64 in 337ms
  creator CPython3Posix(dest=/home/maja/.mozbuild/_virtualenvs/mach, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/maja/.local/share/virtualenv)
    added seed packages: pip==20.2.2, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
running build_ext
copying build/lib.linux-x86_64-3.9/psutil/_psutil_linux.cpython-39-x86_64-linux-gnu.so -> psutil
copying build/lib.linux-x86_64-3.9/psutil/_psutil_posix.cpython-39-x86_64-linux-gnu.so -> psutil

Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
Processing /home/maja/.cache/pip/wheels/0d/f2/30/9639201c1acddc7f0d5ec8b535b8b8f478540542cc8451ab08/zstandard-0.13.0-cp39-cp39-linux_x86_64.whl
Installing collected packages: zstandard
Successfully installed zstandard-0.13.0
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/home/maja/.mozbuild/_virtualenvs/mach/bin/python -m pip install --upgrade pip' command.
Collecting glean_sdk~=32.3.1
  Using cached glean_sdk-32.3.2-cp36-abi3-manylinux1_x86_64.whl (1.5 MB)
Collecting glean-parser==1.28.3
  Using cached glean_parser-1.28.3-py3-none-any.whl (56 kB)
Collecting cffi>=1
  Using cached cffi-1.14.3-cp39-cp39-manylinux1_x86_64.whl (405 kB)
Collecting yamllint>=1.18.0
  Using cached yamllint-1.25.0-py2.py3-none-any.whl (60 kB)
Collecting appdirs>=1.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Processing /home/maja/.cache/pip/wheels/69/60/81/5cd74b8ee068fbe9e04ca0d53148f28f5c6e2c5b177d5dd622/PyYAML-5.3.1-cp39-cp39-linux_x86_64.whl
Collecting diskcache>=4
  Using cached diskcache-5.0.3-py3-none-any.whl (43 kB)
Collecting Click>=7
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting Jinja2>=2.10.1
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting jsonschema>=3.0.2
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting pathspec>=0.5.3
  Using cached pathspec-0.8.0-py2.py3-none-any.whl (28 kB)
Requirement already satisfied: setuptools in /home/maja/.mozbuild/_virtualenvs/mach/lib/python3.9/site-packages (from yamllint>=1.18.0->glean-parser==1.28.3->glean_sdk~=32.3.1) (49.6.0)
Processing /home/maja/.cache/pip/wheels/e0/19/6f/6ba857621f50dc08e084312746ed3ebc14211ba30037d5e44e/MarkupSafe-1.1.1-cp39-cp39-linux_x86_64.whl
Collecting attrs>=17.4.0
  Using cached attrs-20.2.0-py2.py3-none-any.whl (48 kB)
Collecting six>=1.11.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Processing /home/maja/.cache/pip/wheels/00/cc/12/e712d3979a4aeee423b34a832d97c803e1a67659060315a7e0/pyrsistent-0.17.3-cp39-cp39-linux_x86_64.whl
Installing collected packages: pathspec, PyYAML, yamllint, appdirs, diskcache, Click, MarkupSafe, Jinja2, attrs, six, pyrsistent, jsonschema, glean-parser, pycparser, cffi, glean-sdk
Successfully installed Click-7.1.2 Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 appdirs-1.4.4 attrs-20.2.0 cffi-1.14.3 diskcache-5.0.3 glean-parser-1.28.3 glean-sdk-32.3.2 jsonschema-3.2.0 pathspec-0.8.0 pycparser-2.20 pyrsistent-0.17.3 six-1.15.0 yamllint-1.25.0
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/home/maja/.mozbuild/_virtualenvs/mach/bin/python -m pip install --upgrade pip' command.
Python 3 mach environment created.
Traceback (most recent call last):
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv.py", line 46, in <module>
    run()
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv.py", line 42, in run
    run_virtualenv()
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/__main__.py", line 16, in run
    session = cli_run(args, options)
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/run/__init__.py", line 26, in cli_run
    of_session = session_via_cli(args, options, setup_logging)
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/run/__init__.py", line 42, in session_via_cli
    parser, elements = build_parser(args, options, setup_logging)
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/run/__init__.py", line 69, in build_parser
    SeederSelector(interpreter, parser),
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/run/plugin/seeders.py", line 8, in __init__
    possible = self.options("virtualenv.seed")
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 39, in options
    cls._OPTIONS = cls.entry_points_for(key)
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 18, in entry_points_for
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "/home/maja/.pyenv/versions/2.7.18/lib/python2.7/collections.py", line 69, in __init__
    self.__update(*args, **kwds)
  File "/home/maja/.pyenv/versions/2.7.18/lib/python2.7/_abcoll.py", line 571, in update
    for key, value in other:
  File "/home/maja/dev/mozilla-unified/third_party/python/virtualenv/virtualenv/run/plugin/base.py", line 18, in <genexpr>
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "<string>", line 95, in load
  File "/home/maja/.pyenv/versions/2.7.18/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named via_app_data.via_app_data
Error running mach:

    ['create-mach-environment']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file create-mach-environment| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Failed to create virtualenv: /home/maja/.mozbuild/_virtualenvs/mach_py2 (virtualenv.py retcode: 1)

  File "/home/maja/dev/mozilla-unified/python/mozbuild/mozbuild/mach_commands.py", line 1547, in create_mach_environment
    manager.build(sys.executable)
  File "/home/maja/dev/mozilla-unified/python/mozbuild/mozbuild/virtualenv.py", line 490, in build
    self.create(python)
  File "/home/maja/dev/mozilla-unified/python/mozbuild/mozbuild/virtualenv.py", line 266, in create
    self.virtualenv_root, result))
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
WARNING! Failed to create a Python 2 mach environment.

I hope that helps :)

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Hmm, this isn't reproducible for me on Ubuntu with either python 3.8 or 3.9.
I've installed a fresh version of Arch Linux, and that isn't repro-ing the problem, either.

It's worth noting that Maja has encountered the issue in mach create-mach-environment within python 2:

  File "/home/maja/.pyenv/versions/2.7.18/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named via_app_data.via_app_data

while Ivan (the original reporter) had the issue on Python 3:

  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

The next step I'm going to take here is to walk through the virtualenv logic to see if there's anything going on.
Maja, if you're available, it could be useful to do a live Zoom call with a shared screen to see if the issue can be reproduced in a step-by-step way that makes the root cause obvious.

Comment 5

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Sure, feel free to add something to my calendar.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Hey, found how to reproduce the issue: install virtualenv==20.0.0 into the environment that is running ./mach create-mach-environment.
A workaround is to update the virtualenv package that you have installed (ideally to 20.0.31 to match the one vendored in-tree).

:glandium, this is a tricky issue. virtualenv is tripping since it isn't expecting it to be available "twice" in the current environment. It's loading "entry points" and finding those defined in <external environment>/site-packages/virtualenv-$version.dist-info/entry_points.txt and being confused when they don't match what is in the vendored virtualenv package. Off the top of my head and end of my day, I'm not sure what a nice solution to this will be. Do you have any suggestions?

Comment 7

[Frederik Braun [:freddy]]

I have the same error as Maja and others in this bug.
I tried installing virtualenv 20.0.31 globally (pip install virtualenv==20.0.31) and that didn't help.
I noticed that my virtualenv in ~/.mozbuild is a python2 virtualenv but I do have python3 installed. I don't know if that's related.

Happy to help debugging this, if that's helpful. Because I'm blocked on anything else anyway right now.
I am in CEST.

Comment 8

[Mike Hommey [:glandium]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #6)

:glandium, this is a tricky issue. virtualenv is tripping since it isn't expecting it to be available "twice" in the current environment. It's loading "entry points" and finding those defined in <external environment>/site-packages/virtualenv-$version.dist-info/entry_points.txt and being confused when they don't match what is in the vendored virtualenv package. Off the top of my head and end of my day, I'm not sure what a nice solution to this will be. Do you have any suggestions?

My immediate suggestion after trying a few things quickly before my eod/eow, would be to open an issue upstream because it happens the same when running the zipapp (virtualenv.pyz) with a python environment where an older virtualenv is installed.

Comment 9

[Mike Hommey [:glandium]]

Considering their answer to all three existing issues about what is essentially the same problem (https://github.com/pypa/virtualenv/issues?q=is%3Aissue+virtualenv.seed.via_app_data+is%3Aclosed), I'm not optimistic.

Comment 10

[Mike Hommey [:glandium]]

I think I actually found a workaround.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1670039 - Always call virtualenv with python -S.

It disables site modules from the caller python, which are irrelevant
anyways as virtualenv is self-contained.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1654457

Comment 13

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

In case this helps anyone, I was able to fix the issue in my pyenv setup (where create-mach-environment failed to create a Python 2 environment). I just uninstalled virtualenv altogether from my Python 2 environment.

For context, here's my setup again.

(In reply to Maja Frydrychowicz :maja_zf (UTC-4) ([e-mail]) from comment #3)

Ubuntu 20.04.1 LTS

$  python3 --version
Python 3.9.0
$  pyenv global
3.9.0
2.7.18

And pip2.7 freeze showed virtualenv==20.0.14. I tried replacing that with virtualenv==20.0.13 as suggested, but that had no effect.

After running pip2.7 uninstall virtualenv I was able to run ./mach create-mach-environment successfully.

Comment 14

[Mitchell Hentges [:mhentges] 🦀]

Huh, that's bizarre that pip install virtualenv==20.0.31 didn't resolve the issue for :freddy and :maja_zf, sorry about that! Perhaps there was another virtualenv package hiding somewhere that was still throwing it off.
Absolutely, uninstalling virtualenv altogether should work as well.
:glandium's patch looks good to me, it resolves the local reproduce I had.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8f481c06bcd7
Always call virtualenv with python -S. r=firefox-build-system-reviewers,perftest-reviewers,mhentges

Comment 16

[Razvan Maries]

Backed out for build bustages.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=8f481c06bcd7db620ffa51cf4bf06a3003c7574a&searchStr=android%2C4.0%2Capi16%2B%2Cpgo%2Cprofile-guided%2Coptimization%2Cbuilds%2Cgenerate-profile-android-api-16-shippable%2Fopt%2Crun

Log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318225360&repo=autoland&lineNumber=1304

Backout: https://hg.mozilla.org/integration/autoland/rev/9f513b9193508c82686cf4df44a0bda1efd7eed5

Comment 17

[Mike Hommey [:glandium]]

Attachment: Bug 1670039 - Filter-out site-packages and dist-packages from sys.path when running virtualenv.

Comment 18

[Mike Hommey [:glandium]]

Can you check the new patch works for you? (it does for me, but who knows...)

Comment 19

[Francois Guerraz]

Latest patch works for me on Arch.

Comment 20

[Tom S [:evilpie]]

Patch works for me as well. I have the same problem on Linux Mint.

Comment 21

[Frederik Braun [:freddy]]

mach create-mach-environment works for me again. bootstrap still doesn't work as I am now hitting bug 1662632.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/35f6234896f0
Filter-out site-packages and dist-packages from sys.path when running virtualenv. r=firefox-build-system-reviewers,andi,rstewart

Comment 23

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/35f6234896f0

Comment 24

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

wfm

Comment 25

[Mitchell Hentges [:mhentges] 🦀]

Comment on attachment 9180813 [details]
Bug 1670039 - Filter-out site-packages and dist-packages from sys.path when running virtualenv.

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Resolves ./mach bootstrap regression from 1654457
• User impact if declined: ./mach bootstrap may fail depending on users' local python state, unless 1654457 is also declined
• Fix Landed on Version: 83
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Build patch
• String or UUID changes made by this patch:

Comment 26

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9180813 [details]
Bug 1670039 - Filter-out site-packages and dist-packages from sys.path when running virtualenv.

Needed to better support running mach on newer macOS releases. Approved for 78.8esr.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/5e3cb7536f74