1876489 - Perma macosx1015-64 [tier 2] mach.site.InstallPipRequirementsException: Failed to install "/opt/worker/tasks/task_170617638941196/checkouts/gecko/tools/lint/python/black_requirements.txt" into the "python-test" site.

Status: RESOLVED FIXED

(Developer Infrastructure :: Lint and Formatting, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=444558372&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PWBJao6IRB2bQv4HDTD4Sg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-01-25T09:57:22.344Z] Ignoring typed-ast: markers 'python_version < "3.8"' don't match your environment
[task 2024-01-25T09:57:22.344Z] ERROR: Could not find a version that satisfies the requirement black==23.3.0 (from versions: none)
[task 2024-01-25T09:57:22.344Z] ERROR: No matching distribution found for black==23.3.0
[task 2024-01-25T09:57:22.344Z] 
[task 2024-01-25T09:57:22.344Z] Error running mach:
[task 2024-01-25T09:57:22.344Z] 
[task 2024-01-25T09:57:22.344Z]     mach python-test --subsuite mozlint --run-slow
[task 2024-01-25T09:57:22.344Z] 
[task 2024-01-25T09:57:22.344Z] The error occurred in code that was called by the mach command. This is either
[task 2024-01-25T09:57:22.344Z] a bug in the called code itself or in the way that mach is calling it.
[task 2024-01-25T09:57:22.344Z] You can invoke ``./mach busted`` to check if this issue is already on file. If it
[task 2024-01-25T09:57:22.344Z] isn't, please use ``./mach busted file python-test`` to report it. If ``./mach busted`` is
[task 2024-01-25T09:57:22.344Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2024-01-25T09:57:22.344Z] 
[task 2024-01-25T09:57:22.344Z] If filing a bug, please include the full output of mach, including this error
[task 2024-01-25T09:57:22.344Z] message.
[task 2024-01-25T09:57:22.344Z] 
[task 2024-01-25T09:57:22.344Z] The details of the failure are as follows:
[task 2024-01-25T09:57:22.344Z] 
[task 2024-01-25T09:57:22.344Z] mach.site.InstallPipRequirementsException: Failed to install "/opt/worker/tasks/task_170617638941196/checkouts/gecko/tools/lint/python/black_requirements.txt" into the "python-test" site.
[task 2024-01-25T09:57:22.344Z] 
[task 2024-01-25T09:57:22.344Z]   File "/opt/worker/tasks/task_170617638941196/checkouts/gecko/python/mach_commands.py", line 143, in python_test
[task 2024-01-25T09:57:22.344Z]     return run_python_tests(command_context, *args, **kwargs)
[task 2024-01-25T09:57:22.344Z]   File "/opt/worker/tasks/task_170617638941196/checkouts/gecko/python/mach_commands.py", line 206, in run_python_tests
[task 2024-01-25T09:57:22.344Z]     command_context.virtualenv_manager.install_pip_requirements(
[task 2024-01-25T09:57:22.344Z]   File "/opt/worker/tasks/task_170617638941196/checkouts/gecko/python/mach/mach/site.py", line 656, in install_pip_requirements
[task 2024-01-25T09:57:22.344Z]     raise InstallPipRequirementsException(
[taskcluster 2024-01-25T09:57:22.434Z]    Exit Code: 1
[taskcluster 2024-01-25T09:57:22.434Z]    User Time: 6m29.104345s
[taskcluster 2024-01-25T09:57:22.434Z]  Kernel Time: 7m33.109147s
[taskcluster 2024-01-25T09:57:22.434Z]    Wall Time: 3m51.116605s
[taskcluster 2024-01-25T09:57:22.434Z]       Result: FAILED
[taskcluster 2024-01-25T09:57:22.435Z] === Task Finished ===
[taskcluster 2024-01-25T09:57:22.435Z] Task Duration: 3m51.121153s
[taskcluster 2024-01-25T09:57:22.435Z] [mounts] Preserving cache: Moving "/opt/worker/tasks/task_170617638941196/checkouts" to "/opt/worker/cache/RUYmzkrjTkmmcoK5A66b6w"
[taskcluster 2024-01-25T09:57:23.576Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-04-24T09:52:13.480Z
[taskcluster:error] exit status 1

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1876121

:perry.mcmanis, since you are the author of the regressor, bug 1876121, could you take a look?

For more information, please visit BugBot documentation.

Comment 2

[Perry McManis [:perry.mcmanis]]

I am looking at this. Still working on reproducing.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

did python version change, did black requirement change, or did the OS image change?

from what I can tell, black was set in Sept 2023:
https://searchfox.org/mozilla-central/source/tools/lint/python/black_requirements.in#1

We saw in another bug this week where failure to install pytest as it was not found in the cache- not sure if machines were rebuilt or caches were expired.

Comment 5

[Cosmin Sabou [:CosminS]]

This fails only on macosx1015-64 and runs from time to time, I guess when only certain changes happen, TH link.
https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2024-01-19&endday=2024-01-26&tree=trunk&bug=1876489

Comment 6

[Jan-Erik Rediger [:janerik]]

I took a look at Perry's patches and the failure here and also cannot see how this would have been caused by any Glean/Telemetry changes.
We didn't touch tools/lint/python/black_requirements.in, we didn't change much Python except our ours, and the bug 1876121 simply restored functionality that the previous patch removed by mistake.

This failure is also in a completely unrelated tool install.
I also saw this:

[task 2024-01-25T09:57:22.344Z] Looking in links: https://pypi.pub.build.mozilla.org/pub/

Is it loading wheels only from there or is it also contact upstream pypi?

I don't think bug 1876121 is the regressor and I would like someome owning that code look at this.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

we look in our internal pypi https://pypi.pub.build.mozilla.org/pub/ so then we have control over uptime and versions available. :gbrown, do you there there are paths to success by uploading more stuff to https://pypi.pub.build.mozilla.org/pub/ ?

Comment 8

[Geoff Brown [:gbrown]]

Sure, I'll try uploading to the internal pypi to resolve this.

I remain puzzled about how this regressed. It feels like we must have been allowing external pypi downloads previously, but I don't see what has changed.

Comment 9

[Geoff Brown [:gbrown]]

Fixed with many uploads to internal pypi.

I noticed that some requirements are met by packages in third_party/python, but there's no sign of other packages. For instance, codespell is required from https://searchfox.org/mozilla-central/rev/cee2c396081d950f9e3401113fb179999e404ab8/tools/lint/spell/codespell_requirements.txt#7, and that was updated in bug 1860157, but there is no codespell in tree and no attempt was made to add or update codespell in tree in that bug. Hmm...

Comment 10

[Intermittent Failures Robot]

11 failures in 4034 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10
• autoland: 1

Platform and build breakdown:

• macosx1015-64: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1876489&startday=2024-01-22&endday=2024-01-28&tree=all