1721567 - mach build failed: ModuleNotFoundError: No module named 'zstandard'

Status: RESOLVED DUPLICATE of bug 1717884

(Firefox Build System :: General, defect)

Description

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36

Steps to reproduce:

Launch a fresh build on linux-box.
Run command: mach build

Actual results:

Error occurs:

 0:15.05 Error running mach:
 0:15.05     ['--log-no-times', 'artifact', 'toolchain', '--from-build', 'toolchain-linux64-clang']
 0:15.05 The error occurred in code that was called by the mach command. This is either
 0:15.05 a bug in the called code itself or in the way that mach is calling it.
 0:15.05 You can invoke |./mach busted| to check if this issue is already on file. If it
 0:15.05 isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
 0:15.05 misbehaving, you can also inspect the dependencies of bug 1543241.
 0:15.05 If filing a bug, please include the full output of mach, including this error
 0:15.05 message.
 0:15.05 The details of the failure are as follows:
 0:15.05 ModuleNotFoundError: No module named 'zstandard'
 0:15.05   File "/home/justin/FirefoxCode/linux-pc/python/mozbuild/mozbuild/artifact_commands.py", line 571, in artifact_toolchain
 0:15.05     unpack_file(local)
 0:15.05   File "/home/justin/FirefoxCode/linux-pc/python/mozbuild/mozbuild/action/tooltool.py", line 993, in unpack_file
 0:15.05     import zstandard
 0:15.18 ERROR: Command '['/usr/bin/sh', '/home/justin/FirefoxCode/linux-pc/mach', '--log-no-times', 'artifact', 'toolchain', '--from-build', 'toolchain-linux64-clang']' returned non-zero exit status 1.
 0:15.18 ERROR: If you can't fix the above, retry with --disable-bootstrap.
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: ['/usr/bin/python3', '/home/justin/FirefoxCode/linux-pc/configure.py']

  File "/home/justin/FirefoxCode/linux-pc/python/mozbuild/mozbuild/build_commands.py", line 153, in build
    return driver.build(
  File "/home/justin/FirefoxCode/linux-pc/python/mozbuild/mozbuild/controller/building.py", line 1144, in build
    config_rc = self.configure(
  File "/home/justin/FirefoxCode/linux-pc/python/mozbuild/mozbuild/controller/building.py", line 1528, in configure
    status = self._run_command_in_objdir(
  File "/home/justin/FirefoxCode/linux-pc/python/mozbuild/mozbuild/base.py", line 845, in _run_command_in_objdir
    return self.run_process(cwd=self.topobjdir, **args)
  File "/home/justin/FirefoxCode/linux-pc/python/mach/mach/mixin/process.py", line 174, in run_process
    raise Exception(

Expected results:

Build successful.

Comment 1

[Lei Gao]

I think zstandard is missing in third_party/python/requirements.in.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::General' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 3

[Lei Gao]

It's a little hard to append zstandard==0.15.2 to requirements.in, this module needs a platform-specified shared library, we should run setup.py to get it after download.

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Which revision are you on (hg id)? Could you attach the full log for me?

I think zstandard is missing in third_party/python/requirements.in.

third_party/python/requirements.in is for vendored dependencies. zstandard should be downloaded into the mach virtualenv - something isn't working properly on your machine.

Comment 5

[Lei Gao]

Attachment: mach-bootstrap.log

Comment 6

[Lei Gao]

Attachment: mach-build.log

Comment 7

[Lei Gao]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #4)

Which revision are you on (hg id)? Could you attach the full log for me?

df5aed89a479+ tip autoland/central

third_party/python/requirements.in is for vendored dependencies. zstandard should be downloaded into the mach virtualenv - something isn't working properly on your machine.

Attached the command outputs of mach bootstrap and mach build, may helps.

Comment 8

[Mike Hommey [:glandium]]

Can you apply the following patch, run mach build again and attach the new output?

diff --git a/moz.configure b/moz.configure
index 677f51ee3b022..3f3ec7287610c 100755
--- a/moz.configure
+++ b/moz.configure
@@ -327,6 +327,7 @@ def bootstrap_path(path, **kwargs):
             subprocess.run(
                 [
                     shell,
+                    "-x",
                     os.path.join(build_env.topsrcdir, "mach"),
                     "--log-no-times",
                     "artifact",

Comment 9

[Lei Gao]

Attachment: mach-build-verbose.log

Comment 10

[Lei Gao]

(In reply to Mike Hommey [:glandium] from comment #8)

Can you apply the following patch, run mach build again and attach the new output?

Uploaded, see mach-build-verbose.log

I printed the sys.path info also, you can find them at L51 of the attached file.

Comment 11

[Lei Gao]

Duplicate with 1717884, you can close this one.

Comment 12

[Mike Hommey [:glandium]]

Indeed, the output suggests you have MACH_USE_SYSTEM_PYTHON set, in which case you're expected to install the zstandard module yourself.