1843902 - Error building pyyaml when running `./mach vendor python`

Status: RESOLVED FIXED

(Firefox Build System :: Third Party Packaging, defect)

Description

[Andrew Halberstadt [:ahal]]

Recently I've started hitting an error running ./mach vendor python without changes. This isn't tied to any specific revision (it was working last week) and I can reproduce even on mozilla-beta.

STR:

./mach vendor python

Result:

Collecting pyyaml==5.4.1
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [62 lines of output]
      /tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!

              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.

              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/home/ahal/dev/mozilla-unified/third_party/python/pip/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/ahal/dev/mozilla-unified/third_party/python/pip/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/ahal/dev/mozilla-unified/third_party/python/pip/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 314, in run
          self.find_sources()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
          mm.run()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-m1c6maaj/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip

I've reproduced with Python 3.8 through 3.11. So far no luck finding a solution on the internet.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1843902 - Upgrade PyYAML to version 6.0.1, r?ahochheiden!

This also upgrades to Taskgraph 5.6.2 which relaxes its PyYAML constraint to

=5.3.1 to avoid conflicts.

Depends on D181899

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bafc35ec5ecd
Downgrade PyYAML to version 5.3.1, r=ahochheiden

Comment 3

[Cristina Horotan [:chorotan]]

Backed out 2 changesets (bug 1843902, bug 1839891) for causing pyton related build bustages

Backout: https://hg.mozilla.org/integration/autoland/rev/ea2b10290674fae9ab7b34cfbd3e69991b3eb32a

Failure push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=SCVtq7-EQc6pIKhSS421jw.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=bafc35ec5ecd3c25f43751a9d9cac15a3c9ea377

Failure log: https://treeherder.mozilla.org/logviewer?job_id=422901330&repo=autoland&lineNumber=429

Comment 4

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1843902 - Vendor `PyYAML` at version `6.0.1` r?#build

Comment 5

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

It looks the 5.3.1 version of PyYAML wasn't vendoring with a .egg-info and that seems to be what's causing the problems in our CI. I can't explain why it's not a problem locally.

Regardless, there's a fix for the Cython issue in PyYaml 6.0.1, so we don't need to deal with the downgrade, and we can just upgrade instead.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f9efad820be8
Vendor `PyYAML` at version `6.0.1` r=ahal

Comment 7

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/f9efad820be8

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1843902 - Vendor `PyYAML` at version `6.0.1`

There's a Cython issue (https://github.com/yaml/pyyaml/issues/601) with PyYAML that breaks our ./mach vendor python in versions of PyYAML >5.3.1 and =<6.0. This issue has been resolved with PyYAML version 6.0.1, and we can just safely upgrade to it (rather than downgrading to 5.3.1 (which has other issues).

Original Revision: https://phabricator.services.mozilla.com/D183818

Comment 9

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/d29327ad7681