Closed Bug 1686168 Opened 4 months ago Closed 4 months ago

Cannot complete `./mach bootstrap` on macOS Big Sur due to `ModuleNotFoundError: No module named 'zstandard'`

Categories

(Firefox Build System :: Bootstrap Configuration, defect)

x86_64
macOS
defect

Tracking

(firefox-esr78 fixed, firefox86 fixed)

RESOLVED FIXED
86 Branch
Tracking Status
firefox-esr78 --- fixed
firefox86 --- fixed

People

(Reporter: masayuki, Assigned: mhentges)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

 0:06.01 Setting up artifact clang.tar.zst
 0:06.01 Using artifact from local cache: /Users/toybox/.mozbuild/toolchains/650cc3daf2e050e4-clang.tar.zst
Error running mach:

    ['artifact', 'toolchain', '--bootstrap', '--from-build', 'macosx64-clang']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ModuleNotFoundError: No module named 'zstandard'

  File "/Volumes/dev/src/python/mozbuild/mozbuild/artifact_commands.py", line 608, in artifact_toolchain
    unpack_file(local)
  File "/Volumes/dev/src/python/mozbuild/mozbuild/action/tooltool.py", line 993, in unpack_file
    import zstandard

Sentry event ID: d8825ddb5e5a40fbb0deee55717ef153
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
Error running mach:

    ['bootstrap', '--no-interactive']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/Users/toybox/.mozbuild/_virtualenvs/mach/bin/python', '/Volumes/dev/src/mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'macosx64-clang']' returned non-zero exit status 1.

  File "/Volumes/dev/src/python/mozboot/mozboot/mach_commands.py", line 58, in bootstrap
    bootstrapper.bootstrap()
  File "/Volumes/dev/src/python/mozboot/mozboot/bootstrap.py", line 455, in bootstrap
    self.maybe_install_private_packages_or_exit(state_dir, checkout_root)
  File "/Volumes/dev/src/python/mozboot/mozboot/bootstrap.py", line 317, in maybe_install_private_packages_or_exit
    self.instance.ensure_stylo_packages(state_dir, checkout_root)
  File "/Volumes/dev/src/python/mozboot/mozboot/osx.py", line 585, in ensure_stylo_packages
    self.install_toolchain_artifact(state_dir, checkout_root, stylo.MACOS_CLANG)
  File "/Volumes/dev/src/python/mozboot/mozboot/base.py", line 399, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

Sentry event ID: 55b6841f77754245ae348584ef934a6c
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

I tried the workaround commented in bug 1660790 comment 1, bug 1660790 comment 3, but they don't help to avoid this.

Bug 1685006 maybe? There are work arounds for that bug in the comments there.

(In reply to Timothy Nikkel (:tnikkel) from comment #1)

Bug 1685006 maybe? There are work arounds for that bug in the comments there.

Yeah, but ./mach create-mach-environment works in my case. In other words, only the clang setup fails. And bug 1685006 comment 8 didn't help this too.

Hmm, this is strange, you should have the zstandard package installed.
Can you do the following for me?

$ rm -r ~/.mozbuild/_virtualenvs
$ ./mach bootstrap

And paste the full output here? I'm interesting in seeing how bootstrap re-creates your virtualenv :)
Thanks for the report!

Flags: needinfo?(masayuki)

Here is the log. (Choose #2, and then, pressed Enter for the all questions).

Flags: needinfo?(masayuki)

FYI: Probably, this is not a recent regression. I sometimes use macOS for specific bugs, i.e., not using macOS usually.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED

Hmm, that's strange. Mayyybe it's related to your checkout being on a different volume than your primary drive? It's a theory. but I doubt it.

Can you do the following two investigative tasks for me, and comment with the results?

  1. Apply this patch:
diff --git a/python/mozbuild/mozbuild/action/tooltool.py b/python/mozbuild/mozbuild/action/tooltool.py
--- a/python/mozbuild/mozbuild/action/tooltool.py
+++ b/python/mozbuild/mozbuild/action/tooltool.py
@@ -990,6 +990,8 @@ def unpack_file(filename):
         tar.extractall()
         tar.close()
     elif os.path.isfile(filename) and filename.endswith(".tar.zst"):
+        import sys
+        print("1686168: sys.exec={}".format(sys.executable))
         import zstandard
 
         base_file = filename.replace(".tar.zst", "")

Then, do rm -r ~/.mozbuild/_virtualenvs again, then ./mach bootstrap. Finally, submit the output from the bootstrap call.

  1. Try importing zstandard by hand:
    1. source ~/.mozbuild/_virtualenvs/mach/bin/activate
    2. python
    3. At the repl: import zstandard
Flags: needinfo?(masayuki)

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #6)

Hmm, that's strange. Mayyybe it's related to your checkout being on a different volume than your primary drive? It's a theory. but I doubt it.

Ah, I forgot to say it. I need every version environment to test macOS version specific bugs. Therefore, my work area is in an independent volume.

Can you do the following two investigative tasks for me, and comment with the results?

  1. Apply this patch:
diff --git a/python/mozbuild/mozbuild/action/tooltool.py b/python/mozbuild/mozbuild/action/tooltool.py
--- a/python/mozbuild/mozbuild/action/tooltool.py
+++ b/python/mozbuild/mozbuild/action/tooltool.py
@@ -990,6 +990,8 @@ def unpack_file(filename):
         tar.extractall()
         tar.close()
     elif os.path.isfile(filename) and filename.endswith(".tar.zst"):
+        import sys
+        print("1686168: sys.exec={}".format(sys.executable))
         import zstandard
 
         base_file = filename.replace(".tar.zst", "")

Then, do rm -r ~/.mozbuild/_virtualenvs again, then ./mach bootstrap. Finally, submit the output from the bootstrap call.

1686168: sys.exec=/Applications/Xcode.app/Contents/Developer/usr/bin/python3
  1. Try importing zstandard by hand:
    1. source ~/.mozbuild/_virtualenvs/mach/bin/activate
    2. python
    3. At the repl: import zstandard

That does not help me, unfortunately.

Flags: needinfo?(masayuki)

What does ~/.mozbuild/_virtualenvs/mach/bin/python -c 'import sys; print(sys.executable)' say?

Flags: needinfo?(masayuki)

If it says anything else than $HOME/.mozbuild/_virtualenvs/mach/bin/python, there's a problem.

(In reply to Mike Hommey [:glandium] from comment #9)

If it says anything else than $HOME/.mozbuild/_virtualenvs/mach/bin/python, there's a problem.

No, /Users/toybox/.mozbuild/_virtualenvs/mach/bin/python is returned.

Flags: needinfo?(masayuki)

And ~/.mozbuild/_virtualenvs/mach/bin/python -c 'import zstandard' doesn't work?

(In reply to Mike Hommey [:glandium] from comment #11)

And ~/.mozbuild/_virtualenvs/mach/bin/python -c 'import zstandard' doesn't work?

Yeah, ./mach bootstrap after doing it does not work.

Oh, I think this is related to using Xcode python instead of brew python.
To work around this, do brew install python3, then use that Python (until this is fixed).
If you're still running into this with brew's python, please leave a comment :)

Ugh, it's __PYVENV_LAUNCHER__ bugging us again.

Rather than clearing PYVENV_LAUNCHER around problem areas, this
unsets it immediately when mach is launched.

Thanks! The patch works for me, but with requiring rm -r ~/.mozbuild/_virtualenvs and brew uninstall zstd --ignore-dependencies. Or, I succeeded to run ./mach bootstrap without your patch, but setting alias python=/usr/local/bin/python, but anyway, removing the _virtualenvs is required.

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0cdfd2afeefb
Immediately unset __PYVENV_LAUNCHER__ when `mach` starts r=firefox-build-system-reviewers,sheehan,glandium
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch
See Also: → 1693125

Comment on attachment 9197165 [details]
Bug 1686168: Immediately unset PYVENV_LAUNCHER when mach starts

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Resolves regression from 1654457 preventing local builds on Mac with Xcode python.
  • User impact if declined: None
  • Fix Landed on Version: 86
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Build-only patch
  • String or UUID changes made by this patch:
Attachment #9197165 - Flags: approval-mozilla-esr78?

Comment on attachment 9197165 [details]
Bug 1686168: Immediately unset PYVENV_LAUNCHER when mach starts

Approved for 78.8esr.

Attachment #9197165 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+
You need to log in before you can comment on or make changes to this bug.