1869690 - Perma bootstrap ModuleNotFoundError: No module named 'zstandard' - SE

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=439900027&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/G1UoCd82T66HrhhpwmCsBQ/runs/0/artifacts/public/logs/live_backing.log

---

Downloading... 77.6 %
Downloading... 81.3 %
Downloading... 85.0 %
Downloading... 92.4 %
Downloading... 96.1 %
Downloading... 100.0 %
Error running mach:
    mach --log-no-times artifact toolchain --from-task HEkieD-hRgSW0JOglB0AeA:public/build/clang.tar.zst
The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.
If filing a bug, please include the full output of mach, including this error
message.
The details of the failure are as follows:
ModuleNotFoundError: No module named 'zstandard'
  File "/mozilla-unified/python/mozbuild/mozbuild/artifact_commands.py", line 590, in artifact_toolchain
    unpack_file(local)
  File "/mozilla-unified/python/mozbuild/mozbuild/action/tooltool.py", line 1008, in unpack_file
    import zstandard
[taskcluster 2023-10-23 16:59:29.069Z] === Task Finished ===
[taskcluster 2023-10-23 16:59:29.069Z] Unsuccessful task run with exit code: 1 completed in 1332.392 seconds

Comment 1

[Mike Hommey [:glandium]]

Other relevant log that precedes:

ERROR: Exception:
Traceback (most recent call last):
  File "/mozilla-unified/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/mozilla-unified/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/mozilla-unified/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mozilla-unified/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/mozilla-unified/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/mozilla-unified/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/mozilla-unified/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/mozilla-unified/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

IOW, there is a problem with the currently vendored version of pip.

Comment 2

[Mike Hommey [:glandium]]

This is, in fact, the same thing as bug 1857470.

Comment 3

[Mike Hommey [:glandium]]

Actually, let's try fixing this without waiting for a full pip update.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1869690 - Hack the vendored pip's vendored pkg_resources to support python 3.12.

While bug 1857470 is set to fix the problem with a pip upgrade, it is
going to take some time. In the meanwhile, we can just do the minimal
thing that makes things work well enough for a build to go through.

Comment 5

[Serban Stanca [:SerbanS]]

It appears on Fedora as well: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel&searchStr=fedora&revision=b74ed8e9d95a8907add8dd764ccc89be1c7ab99d&selectedTaskRun=VAlNP0cTQp2nmHNlgLS5Vg.0

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a1c0595e579
Hack the vendored pip's vendored pkg_resources to support python 3.12. r=ahochheiden,mach-reviewers

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/2a1c0595e579

Comment 8

[Intermittent Failures Robot]

4 failures in 4224 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• android-5-0-armv7: 2
  • opt: 2
• linux64: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869690&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 9

[Mike Hommey [:glandium]]

Attachment: Bug 1869690 - Hack the vendored pip's vendored pkg_resources to support python 3.12.

While bug 1857470 is set to fix the problem with a pip upgrade, it is
going to take some time. In the meanwhile, we can just do the minimal
thing that makes things work well enough for a build to go through.

Original Revision: https://phabricator.services.mozilla.com/D196256

Comment 10

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/5c983eee5b25