1876121 - `mach bootstrap` broken by Glean update: "AttributeError: module 'pkgutil' has no attribute 'ImpImporter'."

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P1)

Description

[Jim Blandy :jimb]

With Bug 1874923, mach bootstrap no longer runs on Fedora 39 with Python 3.12.1.

central$ hg co 278693c1ad81
obsolete feature not enabled but 248 markers found!
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
central$ rm -rf ~/.mozbuild && hg purge --all && ./mach bootstrap
permanently delete 619 ignored files? (yN) y
Mach and the build system store shared state in a common directory
on the filesystem. The following directory will be created:

  /home/jimb/.mozbuild

If you would like to use a different directory, rename or move it to your
desired location, and set the MOZBUILD_STATE_PATH environment variable
accordingly.
Creating default state directory: /home/jimb/.mozbuild
Creating local state directory: /home/jimb/.mozbuild/srcdirs/central-d3f2b0561fa4
ERROR: Exception:
Traceback (most recent call last):
  File "/home/jimb/moz/central/third_party/python/pip/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/home/jimb/moz/central/third_party/python/pip/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/home/jimb/moz/central/third_party/python/pip/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jimb/moz/central/third_party/python/pip/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/home/jimb/moz/central/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/home/jimb/moz/central/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/home/jimb/moz/central/third_party/python/pip/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/home/jimb/moz/central/third_party/python/pip/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Could not install glean-sdk, so telemetry will not be collected. Continuing.

This is followed by various other errors in the same vein.

Comment 1

[Jim Blandy :jimb]

Confirmed with bisection that this is broken by 278693c1ad81:

central$ hg parent
obsolete feature not enabled but 248 markers found!
changeset:   772282:278693c1ad81
user:        Perry McManis <[e-mail]>
date:        Mon Jan 22 15:45:27 2024 +0000
summary:     Bug 1874923 - Update Glean to 56.1.0 r=janerik,supply-chain-reviewers,mach-reviewers,ahochheiden

Comment 2

[Jim Blandy :jimb]

It seems like Bug 1874923 wiped out the hack to third_party/python/pip/pip/_vendor/pkg_resources/__init__.py introduced in Bug 1869690:

changeset:   768589:2a1c0595e579
user:        Mike Hommey <[e-mail]>
Date:        Wed Dec 13 20:21:49 2023 +0000
summary:     Bug 1869690 - Hack the vendored pip's vendored pkg_resources to support python 3.12. r=ahochheiden,mach-reviewers

	While bug 1857470 is set to fix the problem with a pip upgrade, it is
	going to take some time. In the meanwhile, we can just do the minimal
	thing that makes things work well enough for a build to go through.

	Differential Revision: https://phabricator.services.mozilla.com/D196256

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1874923

:perry.mcmanis, since you are the author of the regressor, bug 1874923, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 4

[Perry McManis [:perry.mcmanis]]

Im going to put the fix back in a patch here. Thanks for alerting us.

Comment 5

[Perry McManis [:perry.mcmanis]]

Attachment: Bug 1876121 - Put back Hack the vendored pip's vendored pkg_resources to support python 3.12 r=travislong!

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7f3e150112c0
Put back Hack the vendored pip's vendored pkg_resources to support python 3.12 r=TravisLong,mach-reviewers,ahochheiden

Comment 7

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/7f3e150112c0

Comment 8

[Perry McManis [:perry.mcmanis]]

Attachment: Bug 1876121 - Put back Hack the vendored pip's vendored pkg_resources to support python 3.12

Original Revision: https://phabricator.services.mozilla.com/D199369

Comment 9

[Phabricator Automation]

Uplift Approval Request

• String changes made/needed: N/A
• Risk associated with taking this patch: Low
• Steps to reproduce for manual QE testing: N/A
• Is Android affected?: yes
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• User impact if declined: This Glean/Glean-Parser update vendoring allows for an expanded number of extra keys, which we were asked to get into 123 by Search.
• Explanation of risk level: This is a vendoring of Glean and Glean-Parser. It also ensures we dont break any changes due to a deprecation of a python package. Most changes were automated by mach vendor, those that arent are small fixes to the resulting code.
• Needs manual QE test: no

Comment 10

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/c51c59c74488