1669934 - Firefox build fails with: ModuleNotFoundError: No module named 'mozversioncontrol'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Jon Coppeard (:jonco)]

mach build fails for me on MacOS with the following output:

 0:02.90 checking for Mercurial version... 5.5.2
 0:03.30 Traceback (most recent call last):
 0:03.30   File "/Users/jon/work/central/configure.py", line 181, in <module>
 0:03.30     sys.exit(main(sys.argv))
 0:03.30   File "/Users/jon/work/central/configure.py", line 53, in main
 0:03.30     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/configure/__init__.py", line 500, in run
 0:03.30     func(*args)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/configure/__init__.py", line 544, in _value_for
 0:03.30     return self._value_for_depends(obj)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/util.py", line 1021, in method_call
 0:03.30     cache[args] = self.func(instance, *args)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/configure/__init__.py", line 553, in _value_for_depends
 0:03.30     value = obj.result()
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/util.py", line 1021, in method_call
 0:03.30     cache[args] = self.func(instance, *args)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
 0:03.30     return self._func(*resolved_args)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/configure/__init__.py", line 1207, in wrapped
 0:03.30     self._apply_imports(func, glob)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/configure/__init__.py", line 880, in _apply_imports
 0:03.30     self._get_one_import(_from, _import, _as, glob)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/configure/__init__.py", line 1010, in _get_one_import
 0:03.30     exec_(import_line, {}, glob)
 0:03.30   File "/Users/jon/work/central/python/mozbuild/mozbuild/util.py", line 56, in exec_
 0:03.30     exec(object, globals, locals)
 0:03.30   File "<string>", line 1, in <module>
 0:03.30 ModuleNotFoundError: No module named 'mozversioncontrol'
 0:03.36 *** Fix above errors and then restart with               "./mach build"
 0:03.36 make: *** [configure] Error 1

I bisected this to bug 1654457.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1654457

Comment 2

[Andrew Halberstadt [:ahal]]

That's really strange..

Could you try running:

$ ./mach clobber python
$ rm -rf <objdir>/_virtualenvs
$ rm -rf ~/.mozbuild/_virtualenvs
$ hg purge --all third_party/python
$ ./mach create-mach-environment

then trying again?

Comment 3

[Andrew Halberstadt [:ahal]]

(aside, we should put above steps into some kind of a generic "my python environment is broken" faq in the source docs)

Comment 4

[Ricky Stewart]

Addendum: rm -rf'ing the virtualenvs before you re-create them should not be necessary. mach create-mach-environment does that for you unless it determines the virtualenvs are already up-to-date. If you want to ignore that logic and unconditionally delete the virtualenvs, you can do mach create-mach-environment -f instead (and probably also file a bug because we'll want to be aware of circumstances where the existing logic isn't sufficient).

Comment 5

[Jon Coppeard (:jonco)]

(In reply to Andrew Halberstadt [:ahal] from comment #2)
OK, I tried that but it still fails.

I tried mach bootstrap but that fails too, with the following output:

~/work/central$ ./mach bootstrap

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
Your choice: 2
Your version of Python 3 (3.8.2) is new enough.
Your version of Python 2 (2.7.17) is new enough.
virtualenv at /Users/jon/.mozbuild/_virtualenvs/mach is already up to date.
Traceback (most recent call last):
  File "/Users/jon/.mozbuild/_virtualenvs/mach/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip._internal.cli.main'
Error running mach:

    ['create-mach-environment']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file create-mach-environment| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/Users/jon/.mozbuild/_virtualenvs/mach/bin/pip', 'install', 'zstandard>=0.9.0,<=0.13.0']' returned non-zero exit status 1.

  File "/Users/jon/work/central/python/mozbuild/mozbuild/mach_commands.py", line 1549, in create_mach_environment
    manager.install_pip_package('zstandard>=0.9.0,<=0.13.0')
  File "/Users/jon/work/central/python/mozbuild/mozbuild/virtualenv.py", line 578, in install_pip_package
    return self._run_pip(args)
  File "/Users/jon/work/central/python/mozbuild/mozbuild/virtualenv.py", line 623, in _run_pip
    subprocess.check_call([pip] + args, stderr=subprocess.STDOUT, cwd=self.topsrcdir,
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/Applications/Xcode.app/Contents/Developer/usr/bin/python3', '/Users/jon/work/central/mach', 'create-mach-environment']' returned non-zero exit status 1.

  File "/Users/jon/work/central/python/mozboot/mozboot/mach_commands.py", line 45, in bootstrap
    bootstrapper.bootstrap()
  File "/Users/jon/work/central/python/mozboot/mozboot/bootstrap.py", line 368, in bootstrap
    self.instance.ensure_mach_environment(checkout_root)
  File "/Users/jon/work/central/python/mozboot/mozboot/base.py", line 277, in ensure_mach_environment
    subprocess.check_call(cmd, cwd=checkout_root)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Oh, interesting, the version of python bundled with Xcode is being used (/Applications/Xcode.app/Contents/Developer/usr/bin/python3).
I gave that a shot myself (/Applications/Xcode.app/.../python3 mach create-mach-environment), but it didn't reproduce the issue, unfortunately.

Jon, can you find some information for me?

• If you source ~/.mozbuild/_virtualenvs/mach/bin/activate and then pip show pip, what version is shown? 20.2.2?
• (while still being source'd into that virtual env) if you run python, then do import pip, import pip._internal, import pip._internal.cli and import pip._internal.cli.main, which one is the first to fail?
• I'm wondering if the bundled python with Xcode is having issues:
  • What is your Xcode version? If you launch Xcode, it should show the version number and build number right under the "Welcome to Xcode" text
  • If you ensure your Xcode is up-to-date via the App Store and try again, can you still reproduce the problem?
• Finally, if these hail-mary questions don't somehow uncover the root of the problem, would you be available during a PST morning/EU end-of-day to do a Zoom call and sort this out live? That might be our best route forward if I can't reproduce this locally.

Comment 7

[Jon Coppeard (:jonco)]

Sure! The python imports all worked:

~$ source ~/.mozbuild/_virtualenvs/mach/bin/activate
(mach) ~$ pip show pip
Name: pip
Version: 20.2.2
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: [e-mail]
License: MIT
Location: /Users/jon/.mozbuild/_virtualenvs/mach/lib/python3.8/site-packages
Requires: 
Required-by: 
(mach) ~$ python
Python 3.8.2 (default, Aug 25 2020, 09:23:57) 
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import pip._internal
>>> import pip._internal.cli
>>> import pip._internal.cli.main

I tried this for the XCode version of python and it does fail:

~$ /Applications/Xcode.app/Contents/Developer/usr/bin/python3
Python 3.8.2 (default, Aug 25 2020, 09:23:57) 
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import pip._internal
>>> import pip._internal.cli
>>> import pip._internal.cli.main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pip._internal.cli.main'

My XCode version is: Version 12.0.1 (12A7300)

I'm very happy to chat on zoom and try and get to the bottom of this.

Comment 8

[Mike Hommey [:glandium]]

Can you try the patch from bug 1670167?

Comment 9

[Jon Coppeard (:jonco)]

(In reply to Mike Hommey [:glandium] from comment #8)
That didn't fix it.

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Perhaps it's from fresh eyes in the morning, or maybe the reboot after my Xcode update changed things, but I'm able to reproduce the issue now with:

/Applications/Xcode.app/Contents/Developer/usr/bin/python3 ./mach create-mach-environment

Workaround

Do you use brew? If so, it could be worth installing python via brew and using that python installation instead. There might be modifications that Xcode has made to its bundled python installation which make it less viable.

If you already have python installed via brew, but the command "which python" still prints the one from Xcode, then would you mind performing the following commands and printing the output here?

$ which python
$ ls -al $(which python)

Next steps for this ticket

I'm going to investigate why the Xcode python is missing pip._internal.cli.main. Perhaps it's workaround-able, or perhaps the Xcode python is stripped in some way and we'll need to update the docs to clarify that it's unusable. I'll take a peek here.
Fortunately, since I've got a local reproduce, we should be ok without a Zoom call unless there's issues running the brew python on your machine, or there's other issues. Let me know!

Comment 11

[Jon Coppeard (:jonco)]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #10)

Do you use brew? If so, it could be worth installing python via brew and using that python installation instead. There might be modifications that Xcode has made to its bundled python installation which make it less viable.

Yes I do. I've never understood whether it's better to install python via brew or use the system one. I'll give this a try.

If you already have python installed via brew, but the command "which python" still prints the one from Xcode, then would you mind performing the following commands and printing the output here?

Looks like python 2 is installed via brew and python 3 is the system one.

~$ which python
/usr/local/bin/python
~$ ls -al $(which python)
lrwxr-x---  1 jon  staff  38  1 Jun 11:43 /usr/local/bin/python -> ../Cellar/python@2/2.7.17_1/bin/python
~$ which python3
/usr/bin/python3
~$ ls -al $(which python3)
-rwxr-xr-x  1 root  wheel  31488 10 Aug 21:56 /usr/bin/python3

Thank you for investigating this!

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

Woah, there's something unexpected going on.
Your python3 is the system python3 (which should be fine, so long as it's not the Xcode one).
However, when you ./mach bootstrap, it seems to be using the Xcode python?

Looks like a zoom call might still be helpful here :)

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

I received some info from :jonco via slack: installing python via brew solved the problem.
Fortunately, that was just the hint I needed to reproduce the "system python => Xcode python" situation:

$ brew uninstall python
$ /usr/bin/python3
Python 3.8.2 (default, Aug 25 2020, 09:23:57) 
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/Applications/Xcode.app/Contents/Developer/usr/bin/python3'

One mystery has been solved!

Comment 14

[Greg Tatum [:gregtatum]]

I'm also running into this issue, and my installation is also mysteriously using the Xcode python.

I've had some random errors around Glean not being installed, and some issues around pyrsistent not being installed. Those issues of mine have been around longer, and I'm not sure it's related. I have some more detailed steps in Bug 1667189 on issues that may or may not be related.

Comment 15

[Greg Tatum [:gregtatum]]

Ok, I fixed it, here was the problem in my case. I have pyenv installed too, which made things a bit more confusing, but essentially here's how to reproduce:

This results in a busted mach build

python --version
> Python 2.7.17

python2 --version
> Python 2.7.17

python3 --version
> Python 3.8.6

This results in a working mach build

python --version
> Python 3.8.6

python2 --version
> Python 2.7.17

python3 --version
> Python 3.8.6

My own personal fix was brew link python3

Comment 16

[Greg Tatum [:gregtatum]]

It seems to me that the build system should be able to detect this issue and deal with it properly, by at the very least throwing an error when we get into this situation with a helpful error message. I made Bug 1667189 depend on this one, since it seems to be the same root issue. I didn't dig further into where the the build system is making this decision, but it's incorrectly assuming python with no numbers is python 3. I believe it should at least check this assumption somehow.

Comment 18

[Ricky Stewart]

(In reply to Greg Tatum [:gregtatum] from comment #16)

I didn't dig further into where the the build system is making this decision, but it's incorrectly assuming python with no numbers is python 3. I believe it should at least check this assumption somehow.

If indeed this is what's going on, this is definitely a bug.

1. We validate that the Python we choose is Python 3 in configure.

2. In the build itself, we go to lengths to always use the pinned Python path from step (1) above and should never just be calling into the python binary blindly.

If you come across more information suggesting where we're doing this, that would be very useful.

Comment 20

[Brad Werth [:bradwerth]]

I had this problem and was able to fix it with the following steps -- I use MacPorts:

1. Uninstall pip from MacPorts: port uninstall py27-pip and port uninstall py38-pip.
2. Ensure that pip and pip3 are mapped to 'none': port select pip none and port select pip3 none.
3. Re-install pip and pip3 in the user directories by downloading get-pip.py and installing with python --user get-pip.py and python3 --user get-pip.py.
4. Ensuring that my shell path includes the user directories.
5. Manually installing glean-sdk with sudo pip3 install -user glean-sdk. Using sudo even though it's installing in the user directory because the install process seems to use root-level temp files, or something.

After all this, and resetting my shell, I could complete mach bootstrap and then mach build.

Comment 21

[Ricky Stewart]

Attachment: Bug 1669934 - Set the `PYTHON3` environment variable where appropriate for the build

This is necessary because at various points we cross over between Python and shell scripts (because the migration in bug 1247781 isn't complete yet). The shell scripts generally use the PYTHON3 environment variable to determine which Python to use. In the absence of this environment variable, it will generally use the Python at the head of the PATH, which is usually the incorrect behavior.

Comment 22

[Ricky Stewart]

Thanks to Greg Tatum, who dug just deep enough to hint to me what was going wrong :)

The patch fixes everything for me. Can someone else confirm the same?

Comment 23

[Mitchell Hentges [:mhentges] 🦀]

Aha, there's nothing wrong with the Xcode python, but it is a little bit "special".
For a full reproduce (on the pip._internal.cli.main error upon create-mach-environment), here's the specific triggers:

• The Xcode python is the default system python
• The Xcode python has pip~=19 installed

1. rm -rf ~/.mozbuild/_virtualenvs (let's clean-slate our mach venvs)
2. ./mach create-mach-environment (successfully creates python environment, installs pip==20, uses that pip to install packages)
3. ./mach create-mach-environment again
   • Fails at around here.
   • It looks like it's calling the pip==20 entrypoint inside the venv using the system python, which then imports the pip==19 package. This then fails, because the pip==20 entrypoint expects pip._internal.cli.main to exist, but the pip==19 package doesn't have it.

I'm putting my money on this having to do with the __PYVENV_LAUNCHER__ environment variable, since brew python doesn't set it, but Xcode python does.

Comment 24

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1669934: Virtualenv should use the correct python packages

Due to the MacOS __PYVENV_LAUNCHER__ environment variable, some
virtualenv operations were being run with the system python (and
packages), rather than the python and packages within the venv.

This was already partially solved by having __PYVENV_LAUNCHER__
get unset when some virtualenv operations were run.
This change makes this more consistent by unsetting the environment
variable once a VirtualenvManager is created.

Comment 25

[Mike Hommey [:glandium]]

(In reply to Jon Coppeard (:jonco) from comment #0)

I bisected this to bug 1654457.

Funnily enough, I bisected it to bug 1670167...

Comment 26

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #25)

(In reply to Jon Coppeard (:jonco) from comment #0)

I bisected this to bug 1654457.

Funnily enough, I bisected it to bug 1670167...

... which is also newer than this bug... O_o

Comment 27

[Mike Hommey [:glandium]]

Mitch was right about __PYVENV_LAUNCHER__, but not at the right place (I think because of the python configure sandboxing of os.environ).

Comment 28

[Mike Hommey [:glandium]]

Attachment: Bug 1669934 - Unset __PYVENV_LAUNCHER__ before reexecuting python configure in the virtualenv.

Comment 29

[Phabricator Automation]

Comment on attachment 9181688 [details]
Bug 1669934 - Set the PYTHON3 environment variable where appropriate for the build

Revision D93606 was moved to bug 1671424. Setting attachment 9181688 [details] to obsolete.

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/553d8ad24a5d
Unset __PYVENV_LAUNCHER__ before reexecuting python configure in the virtualenv. r=firefox-build-system-reviewers,andi,rstewart

Comment 31

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/553d8ad24a5d

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/53e240195131
Virtualenv should use the correct python packages r=firefox-build-system-reviewers,glandium

Comment 33

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/53e240195131

Comment 35

[Mitchell Hentges [:mhentges] 🦀]

Comment on attachment 9181702 [details]
Bug 1669934: Virtualenv should use the correct python packages

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Resolves build-breaking regression from 1654457.
• User impact if declined: Some Mac configurations won't be able to run local builds.
• Fix Landed on Version: 83
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Build patch
• String or UUID changes made by this patch:

Comment 36

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9181702 [details]
Bug 1669934: Virtualenv should use the correct python packages

Needed to better support running mach on newer macOS releases. Approved for 78.8esr.

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/2862a14059a3
https://hg.mozilla.org/releases/mozilla-esr78/rev/366f65b89c37