1739486 - After update to Fedora 36, mach can't create a virtualenv.

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Comment 1

[Emilio Cobos Álvarez (:emilio)]

Apparently the virtualenv python has stopped being in <virtualenv_root>/bin and is now in <virtualenv_root>/local/bin.

Comment 2

[Emilio Cobos Álvarez (:emilio)]

So with a symlink from one place to the other then stuff starts working, but now:

ModuleNotFoundError: No module named 'zstandard'

  File "/home/emilio/src/moz/gecko-2/python/mozbuild/mozbuild/artifact_commands.py", line 556, in artifact_toolchain
    unpack_file(local)
  File "/home/emilio/src/moz/gecko-2/python/mozbuild/mozbuild/action/tooltool.py", line 993, in unpack_file
    import zstandard

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Hmm, try doing a fresh ./mach create-mach-environment -f, then try again, that zstandard issue is floating around and I haven't been able to nail down the cause yet. Are there log lines above of it trying to install zstandard but failing?

I'll dig into this to determine why the virtualenv structure changed.

Comment 4

[Emilio Cobos Álvarez (:emilio)]

So for reference here's the patch I have to apply now:

diff --git a/mach b/mach
index f46c5c4cf4d86..8513d8a78eb23 100755
--- a/mach
+++ b/mach
@@ -102,7 +102,7 @@ if [ -z ${MACH_USE_SYSTEM_PYTHON} ] && [ -z ${MOZ_AUTOMATION} ]
 then
     case "$OSTYPE" in
         cygwin|msys|win32) bin_path=Scripts;;
-        *) bin_path=bin;;
+        *) bin_path=local/bin;;
     esac
     py3executable=$state_dir/_virtualenvs/mach/$bin_path/python
 else
diff --git a/python/mach/mach/virtualenv.py b/python/mach/mach/virtualenv.py
index 0ef14b98ec7d1..5764a58c04bc5 100644
--- a/python/mach/mach/virtualenv.py
+++ b/python/mach/mach/virtualenv.py
@@ -84,7 +84,7 @@ class VirtualenvHelper(object):
         if IS_CYGWIN or IS_NATIVE_WIN:
             return os.path.join(self.virtualenv_root, "Scripts")
 
-        return os.path.join(self.virtualenv_root, "bin")
+        return os.path.join(self.virtualenv_root, "local", "bin")
 
     @property
     def python_path(self):

With that, mach create-mach-environment -f says:

$ ./mach create-mach-environment -f                                                                                                                                                                                                        
created virtual environment CPython3.10.0.final.0-64 in 15ms
  creator CPython3Posix(dest=/home/emilio/.mozbuild/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: glean-sdk==40.0.0 in /home/emilio/.local/lib/python3.10/site-packages (40.0.0)
Requirement already satisfied: glean-parser==3.6.0 in /home/emilio/.local/lib/python3.10/site-packages (from glean-sdk==40.0.0) (3.6.0)
Requirement already satisfied: cffi>=1.13.0 in /usr/lib64/python3.10/site-packages (from glean-sdk==40.0.0) (1.15.0rc2)
Requirement already satisfied: Click>=7 in /usr/lib/python3.10/site-packages (from glean-parser==3.6.0->glean-sdk==40.0.0) (8.0.1)
Requirement already satisfied: diskcache>=4 in /home/emilio/.local/lib/python3.10/site-packages (from glean-parser==3.6.0->glean-sdk==40.0.0) (5.2.1)
Requirement already satisfied: jsonschema<4,>=3.0.2 in /home/emilio/.local/lib/python3.10/site-packages (from glean-parser==3.6.0->glean-sdk==40.0.0) (3.2.0)
Requirement already satisfied: appdirs>=1.4 in /usr/lib/python3.10/site-packages (from glean-parser==3.6.0->glean-sdk==40.0.0) (1.4.4)
Requirement already satisfied: Jinja2>=2.10.1 in /home/emilio/.local/lib/python3.10/site-packages (from glean-parser==3.6.0->glean-sdk==40.0.0) (3.0.2)
Requirement already satisfied: PyYAML>=5.3.1 in /usr/lib64/python3.10/site-packages (from glean-parser==3.6.0->glean-sdk==40.0.0) (6.0)
Requirement already satisfied: yamllint>=1.18.0 in /home/emilio/.local/lib/python3.10/site-packages (from glean-parser==3.6.0->glean-sdk==40.0.0) (1.26.3)
Requirement already satisfied: pycparser in /usr/lib/python3.10/site-packages (from cffi>=1.13.0->glean-sdk==40.0.0) (2.20)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/lib64/python3.10/site-packages (from Jinja2>=2.10.1->glean-parser==3.6.0->glean-sdk==40.0.0) (2.0.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/emilio/.local/lib/python3.10/site-packages (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (0.18.0)
Requirement already satisfied: setuptools in /usr/lib/python3.10/site-packages (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (58.2.0)
Requirement already satisfied: six>=1.11.0 in /usr/lib/python3.10/site-packages (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (1.16.0)
Requirement already satisfied: attrs>=17.4.0 in /home/emilio/.local/lib/python3.10/site-packages (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (21.2.0)
Requirement already satisfied: pathspec>=0.5.3 in /home/emilio/.local/lib/python3.10/site-packages (from yamllint>=1.18.0->glean-parser==3.6.0->glean-sdk==40.0.0) (0.9.0)
Requirement already satisfied: ply==3.11 in /usr/lib/python3.10/site-packages (from pycparser->cffi>=1.13.0->glean-sdk==40.0.0) (3.11)
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: psutil<=5.8.0,>=5.4.2 in /usr/lib64/python3.10/site-packages (5.8.0)
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: zstandard<=0.16.0,>=0.11.1 in /usr/lib64/python3.10/site-packages (0.16.0)
Mach environment created.

Note that "Defaulting to user installation because normal site-packages is not writeable" is very suspicious, why is it looking at /usr?

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739486: Bump "virtualenv" version

Version 20.10.0 adds support for "venv install schemes", which is
needed to avoid unexpected virtualenv directory structures on
environments such as Fedora 36.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

I've got a patch here that should resolve the situation, and it appears to be green so far on try.

Could you try it locally (moz-phab patch --apply-to=here D130404) and see if it works?
My go-to commands for smoke testing virtualenv changes through the various use cases are:

• ./mach create-mach-environment (and running again should intelligently realize that it's already up-to-date)
• ./mach lint -l black --fix
• ./mach configure

Comment 7

[Emilio Cobos Álvarez (:emilio)]

Hmmpf, unfortunately that didn't fix the issue for me, the odd directory structure seems around even after blowing up .mozbuild and co...

Comment 8

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1739486 - Hack around https://github.com/pypa/virtualenv/issues/2208. r=mhentges

The virutalenv hack is in the fedora-distributed version of
virtualenv... Presumably eventually will become unnecessary once they
provide a proper "venv" distutils?

This patch applies both before and after the bump in comment 5, so your
call.

Comment 9

[Emilio Cobos Álvarez (:emilio)]

I reported this in https://bugzilla.redhat.com/show_bug.cgi?id=2020463... The patch above at least allows me to work :-)

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3ae1f4a744f0
Hack around https://github.com/pypa/virtualenv/issues/2208. r=mhentges

Comment 11

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/3ae1f4a744f0