Cannot complete `./mach bootstrap` on macOS Big Sur due to `ModuleNotFoundError: No module named 'zstandard'`
Categories
(Firefox Build System :: Bootstrap Configuration, defect)
Tracking
(firefox-esr78 fixed, firefox86 fixed)
People
(Reporter: masayuki, Assigned: mhentges)
References
(Blocks 1 open bug)
Details
Attachments
(2 files)
|
17.68 KB,
text/plain
|
Details | |
|
48 bytes,
text/x-phabricator-request
|
RyanVM
:
approval-mozilla-esr78+
|
Details | Review |
0:06.01 Setting up artifact clang.tar.zst
0:06.01 Using artifact from local cache: /Users/toybox/.mozbuild/toolchains/650cc3daf2e050e4-clang.tar.zst
Error running mach:
['artifact', 'toolchain', '--bootstrap', '--from-build', 'macosx64-clang']
The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.
If filing a bug, please include the full output of mach, including this error
message.
The details of the failure are as follows:
ModuleNotFoundError: No module named 'zstandard'
File "/Volumes/dev/src/python/mozbuild/mozbuild/artifact_commands.py", line 608, in artifact_toolchain
unpack_file(local)
File "/Volumes/dev/src/python/mozbuild/mozbuild/action/tooltool.py", line 993, in unpack_file
import zstandard
Sentry event ID: d8825ddb5e5a40fbb0deee55717ef153
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
Error running mach:
['bootstrap', '--no-interactive']
The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.
If filing a bug, please include the full output of mach, including this error
message.
The details of the failure are as follows:
subprocess.CalledProcessError: Command '['/Users/toybox/.mozbuild/_virtualenvs/mach/bin/python', '/Volumes/dev/src/mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'macosx64-clang']' returned non-zero exit status 1.
File "/Volumes/dev/src/python/mozboot/mozboot/mach_commands.py", line 58, in bootstrap
bootstrapper.bootstrap()
File "/Volumes/dev/src/python/mozboot/mozboot/bootstrap.py", line 455, in bootstrap
self.maybe_install_private_packages_or_exit(state_dir, checkout_root)
File "/Volumes/dev/src/python/mozboot/mozboot/bootstrap.py", line 317, in maybe_install_private_packages_or_exit
self.instance.ensure_stylo_packages(state_dir, checkout_root)
File "/Volumes/dev/src/python/mozboot/mozboot/osx.py", line 585, in ensure_stylo_packages
self.install_toolchain_artifact(state_dir, checkout_root, stylo.MACOS_CLANG)
File "/Volumes/dev/src/python/mozboot/mozboot/base.py", line 399, in install_toolchain_artifact
subprocess.check_call(cmd, cwd=state_dir)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
Sentry event ID: 55b6841f77754245ae348584ef934a6c
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
I tried the workaround commented in bug 1660790 comment 1, bug 1660790 comment 3, but they don't help to avoid this.
|
||
Comment 1•4 years ago
|
||
Bug 1685006 maybe? There are work arounds for that bug in the comments there.
|
Reporter | |
Comment 2•4 years ago
|
||
(In reply to Timothy Nikkel (:tnikkel) from comment #1)
Bug 1685006 maybe? There are work arounds for that bug in the comments there.
Yeah, but ./mach create-mach-environment works in my case. In other words, only the clang setup fails. And bug 1685006 comment 8 didn't help this too.
|
Assignee | |
Comment 3•4 years ago
|
||
Hmm, this is strange, you should have the zstandard package installed.
Can you do the following for me?
$ rm -r ~/.mozbuild/_virtualenvs
$ ./mach bootstrap
And paste the full output here? I'm interesting in seeing how bootstrap re-creates your virtualenv :)
Thanks for the report!
|
|
Reporter | |
Comment 4•4 years ago
|
||
Here is the log. (Choose #2, and then, pressed Enter for the all questions).
|
|
Reporter | |
Comment 5•4 years ago
|
||
FYI: Probably, this is not a recent regression. I sometimes use macOS for specific bugs, i.e., not using macOS usually.
|
|
Assignee | |
Updated•4 years ago
|
|
|
Assignee | |
Comment 6•4 years ago
|
||
Hmm, that's strange. Mayyybe it's related to your checkout being on a different volume than your primary drive? It's a theory. but I doubt it.
Can you do the following two investigative tasks for me, and comment with the results?
- Apply this patch:
diff --git a/python/mozbuild/mozbuild/action/tooltool.py b/python/mozbuild/mozbuild/action/tooltool.py
--- a/python/mozbuild/mozbuild/action/tooltool.py
+++ b/python/mozbuild/mozbuild/action/tooltool.py
@@ -990,6 +990,8 @@ def unpack_file(filename):
tar.extractall()
tar.close()
elif os.path.isfile(filename) and filename.endswith(".tar.zst"):
+ import sys
+ print("1686168: sys.exec={}".format(sys.executable))
import zstandard
base_file = filename.replace(".tar.zst", "")
Then, do rm -r ~/.mozbuild/_virtualenvs again, then ./mach bootstrap. Finally, submit the output from the bootstrap call.
- Try importing
zstandardby hand:source ~/.mozbuild/_virtualenvs/mach/bin/activatepython- At the repl:
import zstandard
|
|
Reporter | |
Comment 7•4 years ago
|
||
(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #6)
Hmm, that's strange. Mayyybe it's related to your checkout being on a different volume than your primary drive? It's a theory. but I doubt it.
Ah, I forgot to say it. I need every version environment to test macOS version specific bugs. Therefore, my work area is in an independent volume.
Can you do the following two investigative tasks for me, and comment with the results?
- Apply this patch:
diff --git a/python/mozbuild/mozbuild/action/tooltool.py b/python/mozbuild/mozbuild/action/tooltool.py --- a/python/mozbuild/mozbuild/action/tooltool.py +++ b/python/mozbuild/mozbuild/action/tooltool.py @@ -990,6 +990,8 @@ def unpack_file(filename): tar.extractall() tar.close() elif os.path.isfile(filename) and filename.endswith(".tar.zst"): + import sys + print("1686168: sys.exec={}".format(sys.executable)) import zstandard base_file = filename.replace(".tar.zst", "")Then, do
rm -r ~/.mozbuild/_virtualenvsagain, then./mach bootstrap. Finally, submit the output from the bootstrap call.
1686168: sys.exec=/Applications/Xcode.app/Contents/Developer/usr/bin/python3
- Try importing
zstandardby hand:
source ~/.mozbuild/_virtualenvs/mach/bin/activatepython- At the repl:
import zstandard
That does not help me, unfortunately.
|
||
Comment 8•4 years ago
|
||
What does ~/.mozbuild/_virtualenvs/mach/bin/python -c 'import sys; print(sys.executable)' say?
|
|
||
Comment 9•4 years ago
•
|
||
If it says anything else than $HOME/.mozbuild/_virtualenvs/mach/bin/python, there's a problem.
|
|
Reporter | |
Comment 10•4 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #9)
If it says anything else than $HOME/.mozbuild/_virtualenvs/mach/bin/python, there's a problem.
No, /Users/toybox/.mozbuild/_virtualenvs/mach/bin/python is returned.
|
|
||
Comment 11•4 years ago
|
||
And ~/.mozbuild/_virtualenvs/mach/bin/python -c 'import zstandard' doesn't work?
|
|
Reporter | |
Comment 12•4 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #11)
And
~/.mozbuild/_virtualenvs/mach/bin/python -c 'import zstandard'doesn't work?
Yeah, ./mach bootstrap after doing it does not work.
|
|
Assignee | |
Comment 13•4 years ago
|
||
Oh, I think this is related to using Xcode python instead of brew python.
To work around this, do brew install python3, then use that Python (until this is fixed).
If you're still running into this with brew's python, please leave a comment :)
|
|
Assignee | |
Comment 14•4 years ago
|
||
Ugh, it's __PYVENV_LAUNCHER__ bugging us again.
|
|
Assignee | |
Comment 15•4 years ago
|
||
Rather than clearing PYVENV_LAUNCHER around problem areas, this
unsets it immediately when mach is launched.
|
|
Reporter | |
Comment 16•4 years ago
|
||
Thanks! The patch works for me, but with requiring rm -r ~/.mozbuild/_virtualenvs and brew uninstall zstd --ignore-dependencies. Or, I succeeded to run ./mach bootstrap without your patch, but setting alias python=/usr/local/bin/python, but anyway, removing the _virtualenvs is required.
|
||
Comment 17•4 years ago
|
||
|
||
Comment 18•4 years ago
|
||
| bugherder | ||
|
|
Assignee | |
Comment 19•4 years ago
|
||
Comment on attachment 9197165 [details]
Bug 1686168: Immediately unset PYVENV_LAUNCHER when mach starts
ESR Uplift Approval Request
- If this is not a sec:{high,crit} bug, please state case for ESR consideration: Resolves regression from 1654457 preventing local builds on Mac with Xcode python.
- User impact if declined: None
- Fix Landed on Version: 86
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): Build-only patch
- String or UUID changes made by this patch:
|
||
Comment 20•4 years ago
|
||
Comment on attachment 9197165 [details]
Bug 1686168: Immediately unset PYVENV_LAUNCHER when mach starts
Approved for 78.8esr.
|
|
||
Comment 21•4 years ago
|
||
| uplift | ||
Description
•