1738848 - Windows 10+VS2019 build error: `./mach bootstrap` fails with "ModuleNotFoundError: No module named 'zstandard'"

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[ADing]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36 Edg/95.0.1020.40
Firefox for Android

Steps to reproduce:

I'm trying to build Firefox on Windows 10 using Visual Studio 2019.
The latest version of MozillaBuild has been installed at "C:\mozilla-build".
The source code has been cloned into "C:\mozilla-source" by using this command: "hg clone https://hg.mozilla.org/mozilla-central".
When I run "cd C:/mozilla-source/mozilla-central" and "mach bootstrap", it fails with "ModuleNotFoundError: No module named 'zstandard'".
I have tried Bug 1686168 and Bug 1660790, they all does not work for me.
The dictionary "~/.mozbuild/_virtualenvs/mach" has no subfolder named "bin".

Comment 1

[ADing]

$ mach bootstrap
Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode [default]
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
  5. SpiderMonkey JavaScript engine
Your choice:2
virtualenv at c:/Users/admin/.mozbuild\_virtualenvs\mach is already up to date.
Mach environment created.
Your version of Mercurial (5.1.2) is sufficiently modern.
Your version of Rust (1.56.1) is new enough.
Rust supports x86_64-pc-windows-msvc targets.
info: downloading component 'rust-std' for 'i686-pc-windows-msvc'
info: installing component 'rust-std' for 'i686-pc-windows-msvc'
info: using up to 500.0 MiB of RAM to unpack components

Mozilla recommends a number of changes to Mercurial to enhance your
experience with it.

Would you like to run a configuration wizard to ensure Mercurial is
optimally configured? (Yn): Y
================================================================================
Ensuring https://hg.mozilla.org/hgcustom/version-control-tools is up to date at c:/Users/admin/.mozbuild\version-control-tools
pulling from https://hg.mozilla.org/hgcustom/version-control-tools
searching for changes
no changes found
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
================================================================================
This wizard will guide you through configuring Mercurial for an optimal
experience contributing to Mozilla projects.

The wizard makes no changes without your permission.

To begin, press the enter/return key.
 \

It looks like the setup wizard has already installed a copy of the
evolve extension on your machine, at c:\Users\admin\.mozbuild/evolve.

(Relevant config option: extensions.evolve)

Would you like to update evolve to the latest version?  (Yn)  Y
pulling from https://www.mercurial-scm.org/repo/evolve/
no changes found
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Evolve was updated successfully.
 0:15.17 Setting up artifact node.tar.zst
 0:15.18 Using artifact from local cache: c:\Users\admin\.mozbuild\toolchains\a9c74a6a69012027-node.tar.zst
Error running mach:

    ['artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ModuleNotFoundError: No module named 'zstandard'

  File "c:\mozilla-source\mozilla-central\python/mozbuild/mozbuild/artifact_commands.py", line 556, in artifact_toolchain
    unpack_file(local)
  File "c:\mozilla-source\mozilla-central\python/mozbuild\mozbuild\action\tooltool.py", line 993, in unpack_file
    import zstandard
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['c:/Users/admin/.mozbuild\\_virtualenvs\\mach\\Scripts\\python.exe', 'c:\\mozilla-source\\mozilla-central\\mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'win64-node']' returned non-zero exit status 1.

  File "c:\mozilla-source\mozilla-central\python/mozboot/mozboot/mach_commands.py", line 42, in bootstrap
    bootstrapper.bootstrap(command_context.settings)
  File "c:\mozilla-source\mozilla-central\python/mozboot\mozboot\bootstrap.py", line 377, in bootstrap
    state_dir, checkout_root, application
  File "c:\mozilla-source\mozilla-central\python/mozboot\mozboot\bootstrap.py", line 224, in maybe_install_private_packages_or_exit
    self.instance.ensure_node_packages(state_dir, checkout_root)
  File "c:\mozilla-source\mozilla-central\python/mozboot\mozboot\mozillabuild.py", line 245, in ensure_node_packages
    self.install_toolchain_artifact(state_dir, checkout_root, node_artifact)
  File "c:\mozilla-source\mozilla-central\python/mozboot\mozboot\base.py", line 428, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "c:\mozilla-build\python3\lib\subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Bootstrap Configuration' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Thanks for the report 👍
Can you show me the output of ./mach create-mach-environment -f?

Comment 4

[ADing]

Attachment: Result of command "./mach create-mach-environment -f"

Comment 5

[ADing]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #3)

Thanks for the report 👍
Can you show me the output of ./mach create-mach-environment -f?

Thanks for your reply. The attachment is the result of command "./mach create-mach-environment -f".
BTW. This error message occurs too when I first run the command "./mach bootstrap".

Comment 6

[James Teh [:Jamie]]

I just saw this too when updating today despite having a build environment that was working just fine last week. For me, though, removing ~/.mozbuild/_virtualenvs and re-running mach bootstrap fixed it.

What's weird is that before I did that, running the python in the venv and doing import zstandard wasn't failing!

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

ADing, looks like you have a separate issue preventing you from doing a ./mach create-mach-environment successfully. Specifically:

ValueError: check_hostname requires server_hostname

Have you configured a proxy? That seems to be the cause of the issue. For example, do you have the http_proxy or https_proxy environment variable set? Can you share the results of the env command?

---

Related: it's unclear why the recent changes are requiring a re-creation of the Mach virtualenv. Especially (and thanks for sharing this, James):

What's weird is that before I did that, running the python in the venv and doing import zstandard wasn't failing!

I'm trying to reproduce locally here:

1. Clean checkout at 02ea2d15bf4c17c346aa9e06775c7027cdfacabf, before the recent regressing landings
2. Purge obj-*, ~/.mozbuild/_virtualenvs
3. ./mach bootstrap with artifact extraction
4. Update to 0b166cabc98d7edf925b3d1dd000b59d4028752e, after the regressing landings
5. ./mach bootstrap with artifact extraction

I'm not getting the zstandard import error 😔
EDIT: I have a home machine dev machine running Windows that I haven't updated to newer m-c in a while, I'll see if I can repro there to get to the root cause.

Comment 8

[ADing]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #7)

ADing, looks like you have a separate issue preventing you from doing a ./mach create-mach-environment successfully. Specifically:

ValueError: check_hostname requires server_hostname

Have you configured a proxy? That seems to be the cause of the issue. For example, do you have the http_proxy or https_proxy environment variable set? Can you share the results of the env command?

---

Related: it's unclear why the recent changes are requiring a re-creation of the Mach virtualenv. Especially (and thanks for sharing this, James):

What's weird is that before I did that, running the python in the venv and doing import zstandard wasn't failing!

I'm trying to reproduce locally here:

1. Clean checkout at 02ea2d15bf4c17c346aa9e06775c7027cdfacabf, before the recent regressing landings
2. Purge obj-*, ~/.mozbuild/_virtualenvs
3. ./mach bootstrap with artifact extraction
4. Update to 0b166cabc98d7edf925b3d1dd000b59d4028752e, after the regressing landings
5. ./mach bootstrap with artifact extraction

I'm not getting the zstandard import error 😔
EDIT: I have a home machine dev machine running Windows that I haven't updated to newer m-c in a while, I'll see if I can repro there to get to the root cause.

Yeah, I have configured a HTTP proxy in operating system to accelerate the downloading process.
And bootstrap works well after I close the HTTP proxy.
The command "./mach build" also runs successfully now.
In a word, the HTTP proxy is the root of this problem which tortured me for several days.
Thank you so much. I will close this issue.
:)

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Aha, I've discovered the root cause.
We identify whether we're running in a "Mach" virtualenv or not based on whether it has a moz_virtualenv_metadata.json file.
If the file doesn't exist, we can't tell whether it's because:

• The current virtualenv isn't related to Mach (e.g.: it's a user-managed, external virtualenv), or
• The current virtualenv is managed by Mach, but it's old and doesn't yet have the moz_virtualenv_metadata.json file

We used to do this "am I in a Mach virtualenv" identification by simply checking if we're in any virtualenv.
The migration path here should be improved.

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1738848: Handle virtualenv case where metadata file is missing

At Mach initialization time, it needs to know if it's not running from
the Mach virtualenv so it can decide to insulate itself from the
external sys.path.

However, the current mechanism of detecting the Mach virtualenv
struggles if the venv is old, and therefore missing its metadata file.
In such a case, it's recognized as "not the Mach venv" rather than "an
out-of-date Mach venv".

As part of this, I'm realizing that simply using the metadata file as a
"virtualenv has finished building" marker is insufficient, because it
will cause similar confusion here. This is solved with a "finalization"
key in in the metadata.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e77052ebc4ae
Handle virtualenv case where metadata file is missing r=ahal

Comment 12

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/e77052ebc4ae