1751284 - mach bootstrap fails in a clean ubuntu 21.10 installation with ImportError: cannot import name 'dist' from 'distutils' (/usr/lib/python3.9/distutils/__init__.py)

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect, P5)

Comment 1

[Emilio Cobos Álvarez (:emilio)]

Per this comment this is not supposed to happen. Perhaps a recent regression?

Comment 2

[Emilio Cobos Álvarez (:emilio)]

Whole stack for reference:

  File "/home/emilio/src/moz/gecko/./mach", line 96, in <module>
    main(sys.argv[1:])
  File "/home/emilio/src/moz/gecko/./mach", line 88, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "/home/emilio/src/moz/gecko/./mach", line 35, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/home/emilio/src/moz/gecko/./mach", line 21, in load_mach
    return mach_initialize.initialize(dir_path)
  File "/home/emilio/src/moz/gecko/build/mach_initialize.py", line 212, in initialize
    _activate_python_environment(
  File "/home/emilio/src/moz/gecko/build/mach_initialize.py", line 151, in _activate_python_environment
    mach_environment.activate()
  File "/home/emilio/src/moz/gecko/python/mach/mach/site.py", line 320, in activate
    self.ensure()
  File "/home/emilio/src/moz/gecko/python/mach/mach/site.py", line 305, in ensure
    up_to_date = self.up_to_date()
  File "/home/emilio/src/moz/gecko/python/mach/mach/site.py", line 299, in up_to_date
    self._pthfile_lines(environment),
  File "/home/emilio/src/moz/gecko/python/mach/mach/site.py", line 378, in _pthfile_lines
    *_deprioritize_venv_packages(environment),
  File "/home/emilio/src/moz/gecko/python/mach/mach/site.py", line 1035, in _deprioritize_venv_packages
    virtualenv.site_packages_dir(),
  File "/home/emilio/src/moz/gecko/python/mach/mach/site.py", line 710, in site_packages_dir
    from distutils import dist
ImportError: cannot import name 'dist' from 'distutils' (/usr/lib/python3.9/distutils/__init__.py)

Comment 3

[Emilio Cobos Álvarez (:emilio)]

After installing python3-distutils, I can progress till:

created virtual environment CPython3.9.7.final.0-64 in 23ms
  creator CPython3Posix(dest=/home/emilio/.mozbuild/srcdirs/mozilla-central-e0ec58a5ed0d/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Collecting glean-sdk==43.0.2
  Downloading glean_sdk-43.0.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 1.9 MB/s 
Collecting cffi>=1.13.0
  Downloading cffi-1.15.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (444 kB)
     |████████████████████████████████| 444 kB 7.5 MB/s 
Collecting glean-parser==4.4.0
  Downloading glean_parser-4.4.0-py3-none-any.whl (86 kB)
     |████████████████████████████████| 86 kB 3.5 MB/s 
Requirement already satisfied: PyYAML>=5.3.1 in ./third_party/python/PyYAML/lib3 (from glean-parser==4.4.0->glean-sdk==43.0.2) (5.4.1)
Requirement already satisfied: appdirs>=1.4 in ./third_party/python/appdirs (from glean-parser==4.4.0->glean-sdk==43.0.2) (1.4.4)
Requirement already satisfied: jsonschema>=3.0.2 in ./third_party/python/jsonschema (from glean-parser==4.4.0->glean-sdk==43.0.2) (3.2.0)
Requirement already satisfied: Click>=7 in ./third_party/python/click (from glean-parser==4.4.0->glean-sdk==43.0.2) (7.1.2)
Requirement already satisfied: diskcache>=4 in ./third_party/python/diskcache (from glean-parser==4.4.0->glean-sdk==43.0.2) (4.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in ./third_party/python/Jinja2 (from glean-parser==4.4.0->glean-sdk==43.0.2) (2.11.3)
Requirement already satisfied: yamllint>=1.18.0 in ./third_party/python/yamllint (from glean-parser==4.4.0->glean-sdk==43.0.2) (1.23.0)
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     |████████████████████████████████| 118 kB 8.3 MB/s 
Requirement already satisfied: MarkupSafe>=0.23 in ./third_party/python/MarkupSafe/src (from Jinja2>=2.10.1->glean-parser==4.4.0->glean-sdk==43.0.2) (1.1.1)
Requirement already satisfied: pyrsistent>=0.14.0 in ./third_party/python/pyrsistent (from jsonschema>=3.0.2->glean-parser==4.4.0->glean-sdk==43.0.2) (0.16.0)
Requirement already satisfied: six>=1.11.0 in ./third_party/python/six (from jsonschema>=3.0.2->glean-parser==4.4.0->glean-sdk==43.0.2) (1.13.0)
Requirement already satisfied: setuptools in ./third_party/python/setuptools (from jsonschema>=3.0.2->glean-parser==4.4.0->glean-sdk==43.0.2) (51.2.0)
Requirement already satisfied: attrs>=17.4.0 in ./third_party/python/attrs (from jsonschema>=3.0.2->glean-parser==4.4.0->glean-sdk==43.0.2) (19.1.0)
Requirement already satisfied: pathspec>=0.5.3 in ./third_party/python/pathspec (from yamllint>=1.18.0->glean-parser==4.4.0->glean-sdk==43.0.2) (0.9.0)
Installing collected packages: pycparser, glean-parser, cffi, glean-sdk
Successfully installed cffi-1.15.0 glean-parser-4.4.0 glean-sdk-43.0.2 pycparser-2.21
Collecting psutil<=5.8.0,>=5.4.2
  Downloading psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl (293 kB)
     |████████████████████████████████| 293 kB 1.9 MB/s 
Installing collected packages: psutil
Successfully installed psutil-5.8.0
Collecting zstandard<=0.16.0,>=0.11.1
  Downloading zstandard-0.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB)
     |████████████████████████████████| 2.9 MB 1.8 MB/s 
Installing collected packages: zstandard
Successfully installed zstandard-0.16.0
Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode [default]
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
  5. SpiderMonkey JavaScript engine
Your choice:2
ERROR: Could not find pip3.
HINT: Try installing pip3 with `apt-get install python3-pip`.
ERROR: Your Python installation will not be able to run `mach bootstrap`. `mach bootstrap` cannot maintain your Python environment for you; fix the errors shown here, and then re-run `mach bootstrap`.

Which is a much nicer error message :)

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

These errors are printed in common Mach code before bootstrap is running, and platform-specific checks for prerequisites all exist in bootstrap (and need Mach to have finished initializing).

Our best bet to solve this problem is via docs, and we do already: we request the installation of python3-pip. which will automatically install python3-distutils.

I suppose we can get fancier here and do a try import distutils, except "please install distutils on your machine" in the top level ./mach file, but we're going to have diminishing returns.

(Note: I think it's ok to leave the python3-pip check to bootstrap, which will be able to provide a more tailored error message).

I'll leave this as a good-first-bug.

Good First Bug summary

After the version check, check if distutils can be imported. If not, print the error:

Mach needs the Python "distutils" module, but it's not available on your machine. This is most common on Debian Linux (or derivatives), and can be fixed by installing a package named similiar to `python3-distutils`.

Comment 5

[Mike Hommey [:glandium]]

Ironically, there's this comment above the from distutils import dist:

# Defer "distutils" import until this function is called so that
# "mach bootstrap" doesn't fail due to Linux distro python-distutils
# package not being installed.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Heh, I suppose that comment has been obsolete since Mach has started automatically managing its own virtualenv.

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Hmm, those obsolete docs are going to bug me, I'll resolve this.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1751284: Assert for `distutils` during Mach initialization

Our platform-specific bootstrap code transitively depends on
the standard library's distutils module, but Debian Linux and derived
distros generally split distutils into a separate installable OS
package from python3 itself.

So, Python 3 being able to run isn't a sufficient guarantee that
distutils is available.

To catch this case, add a distutils check at the very beginning of
Mach initialization.

---

Also remove an obsolete comment claiming that mach bootstrap doesn't
need distutils, which was obsoleted by Bug 1717051.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7486109751e9
Assert for `distutils` during Mach initialization r=ahal

Comment 10

[Kagami Rosylight [:saschanaz] (they/them)]

This is causing a deprecation warning for every mach run:

> ./mach try chooser
C:\Users\sasch\Documents\GitHub\gecko-dev\mach:81: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils
No broken requirements found.
Task configuration changed, generating target task set
Starting trychooser on http://127.0.0.1:5000
 * Serving Flask app "tryselect.selectors.chooser.app" (lazy loading)
 * Environment: development
 * Debug mode: off

Comment 11

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/7486109751e9

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

Hmm, true, I'll bump the prioritization on migrating away from distutils.