1769237 - Can't build on Arch Linux anymore (because of old packaging vendored package?)

Status: RESOLVED DUPLICATE of bug 1769205

(Firefox Build System :: Mach Core, defect)

Description

[Emilio Cobos Álvarez (:emilio)]

After updating to the latest python-setuptools provided by my distro, I can't build anymore:

Traceback (most recent call last):
  File "/home/emilio/src/moz/gecko-4/mach", line 96, in <module>
    main(sys.argv[1:])
  File "/home/emilio/src/moz/gecko-4/mach", line 88, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "/home/emilio/src/moz/gecko-4/mach", line 35, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/home/emilio/src/moz/gecko-4/mach", line 21, in load_mach
    return mach_initialize.initialize(dir_path)
  File "/home/emilio/src/moz/gecko-4/build/mach_initialize.py", line 153, in initialize
    _activate_python_environment(
  File "/home/emilio/src/moz/gecko-4/build/mach_initialize.py", line 93, in _activate_python_environment
    from mach.site import MachSiteManager
  File "/home/emilio/src/moz/gecko-4/python/mach/mach/site.py", line 26, in <module>
    from mach.requirements import (
  File "/home/emilio/src/moz/gecko-4/python/mach/mach/requirements.py", line 7, in <module>
    from packaging.requirements import Requirement
  File "/home/emilio/src/moz/gecko-4/third_party/python/packaging/packaging/requirements.py", line 24, in <module>
    from .markers import MARKER_EXPR, Marker
  File "/home/emilio/src/moz/gecko-4/third_party/python/packaging/packaging/markers.py", line 25, in <module>
    from .specifiers import InvalidSpecifier, Specifier
  File "/home/emilio/src/moz/gecko-4/third_party/python/packaging/packaging/specifiers.py", line 14, in <module>
    from .utils import canonicalize_version
  File "/home/emilio/src/moz/gecko-4/third_party/python/packaging/packaging/utils.py", line 9, in <module>
    from .tags import Tag, parse_tag
  File "/home/emilio/src/moz/gecko-4/third_party/python/packaging/packaging/tags.py", line 7, in <module>
    import distutils.util
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "/usr/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 92, in create_module
    return importlib.import_module('setuptools._distutils')
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 16, in <module>
    import setuptools.version
  File "/usr/lib/python3.10/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3102, in <module>
    class RequirementParseError(packaging.requirements.InvalidRequirement):

It seems like a bad interaction between the vendored third_party/python/packaging/packaging and built-in pkg_resources which uses packaging, but expects a newer version.

Andrew, you seem familiar with the whole python vendoring situation, do you know what the right thing to do is? It seems weird that the vendored packaging ends up depending on system pkg_resources which depends on a newer packaging module. Is there any easy way to update vendored packaging or other things that could fix this?

I tried MACH_USE_SYSTEM_PYTHON but that doesn't seem to help.

Comment 1

[Jan Alexander Steffens [:heftig]]

FWIW, MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system doesn't help, either.

Comment 3

[Andrew Halberstadt [:ahal]]

Oof, I did review a bunch of Mitch's patches here, but I had at best second hand understanding of how it works. Now that Mitch has left there aren't any experts in this area. I'll leave the needinfo for now and try to do a bit of digging next week, but can't promise anything.

Issues like this belong to the build team, but given Mitch's recent departure we'll need to train someone else up in this area. I'll bring this up with :glob.

Comment 4

[Andrew Halberstadt [:ahal]]

Adding needinfo to Alex in case he knows how to investigate.

Comment 5

[Emilio Cobos Álvarez (:emilio)]

In bug 1769205 there's a tentative patch, but I'm not sure if it's reasonable?

Comment 6

[Alex Hochheiden [:ahochheiden]]

(In reply to Andrew Halberstadt [:ahal] from comment #4)

Adding needinfo to Alex in case he knows how to investigate.

I also don't know that much about how all this works, but I'll put in some time right now to get a better understanding.