1879665 - Perma OSX/Linux 10.15 WebRender Could not install python package: failed all attempts.

Status: RESOLVED FIXED

(Testing :: CI Configuration, defect)

Description

[Treeherder Bug Filer]

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=446747737&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/eRldTU2LSTiLeo3bY6kzTQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-02-09T22:19:37.415Z] 22:19:37     INFO - Using env: (same as previous command)
[task 2024-02-09T22:19:37.836Z] 22:19:37     INFO -  Looking in links: https://pypi.pub.build.mozilla.org/pub/, https://pypi.pub.build.mozilla.org/pub/
[task 2024-02-09T22:19:37.837Z] 22:19:37     INFO -  Requirement already satisfied: aioquic==0.9.21 in /opt/worker/tasks/task_170751661357128/build/venv/lib/python3.7/site-packages (from -r /opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport/requirements.txt (line 1)) (0.9.21)
[task 2024-02-09T22:19:37.843Z] 22:19:37     INFO -  Requirement already satisfied: certifi in /opt/worker/tasks/task_170751661357128/build/venv/lib/python3.7/site-packages (from aioquic==0.9.21->-r /opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport/requirements.txt (line 1)) (2018.4.16)
[task 2024-02-09T22:19:38.430Z] 22:19:38     INFO -  Collecting pyopenssl>=22
[task 2024-02-09T22:19:38.438Z] 22:19:38     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/pyOpenSSL-23.2.0-py3-none-any.whl (59 kB)
[task 2024-02-09T22:19:38.548Z] 22:19:38     INFO -  Collecting pylsqpack<0.4.0,>=0.3.3
[task 2024-02-09T22:19:38.554Z] 22:19:38     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/pylsqpack-0.3.17-cp37-cp37m-macosx_10_9_x86_64.whl (161 kB)
[task 2024-02-09T22:19:38.673Z] 22:19:38     INFO -  Collecting cryptography!=40.0.0,!=40.0.1,<42,>=38.0.0
[task 2024-02-09T22:19:38.694Z] 22:19:38     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/cryptography-41.0.2-cp37-abi3-macosx_10_12_x86_64.whl (2.8 MB)
[task 2024-02-09T22:19:38.849Z] 22:19:38     INFO -  Collecting cffi>=1.12
[task 2024-02-09T22:19:38.858Z] 22:19:38     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/cffi-1.16.0.tar.gz (512 kB)
[task 2024-02-09T22:19:38.977Z] 22:19:38     INFO -    Installing build dependencies: started
[task 2024-02-09T22:19:40.040Z] 22:19:40     INFO -    Installing build dependencies: finished with status 'error'
[task 2024-02-09T22:19:40.047Z] 22:19:40     INFO -    error: subprocess-exited-with-error
[task 2024-02-09T22:19:40.047Z] 22:19:40     INFO -    × pip subprocess to install build dependencies did not run successfully.
[task 2024-02-09T22:19:40.047Z] 22:19:40     INFO -    │ exit code: 1
[task 2024-02-09T22:19:40.047Z] 22:19:40     INFO -    ╰─> [3 lines of output]
[task 2024-02-09T22:19:40.048Z] 22:19:40     INFO -        Looking in links: https://pypi.pub.build.mozilla.org/pub/, https://pypi.pub.build.mozilla.org/pub/, https://pypi.pub.build.mozilla.org/pub/
[task 2024-02-09T22:19:40.048Z] 22:19:40     INFO -        ERROR: Could not find a version that satisfies the requirement setuptools>=66.1 (from versions: rust-0.12.1, 4.0.1, 41.2.0, 43.0.0, 57.0.0, 59.2.0)
[task 2024-02-09T22:19:40.048Z] 22:19:40     INFO -        ERROR: No matching distribution found for setuptools>=66.1
[task 2024-02-09T22:19:40.048Z] 22:19:40     INFO -        [end of output]
[task 2024-02-09T22:19:40.048Z] 22:19:40     INFO -    note: This error originates from a subprocess, and is likely not a problem with pip.
[task 2024-02-09T22:19:40.049Z] 22:19:40     INFO -  error: subprocess-exited-with-error
[task 2024-02-09T22:19:40.049Z] 22:19:40     INFO -  × pip subprocess to install build dependencies did not run successfully.
[task 2024-02-09T22:19:40.049Z] 22:19:40     INFO -  │ exit code: 1
[task 2024-02-09T22:19:40.049Z] 22:19:40     INFO -  ╰─> See above for output.
[task 2024-02-09T22:19:40.050Z] 22:19:40     INFO -  note: This error originates from a subprocess, and is likely not a problem with pip.
[task 2024-02-09T22:19:40.139Z] 22:19:40     INFO - Return code: 1
[task 2024-02-09T22:19:40.139Z] 22:19:40     INFO - retry: Failed, sleeping 300 seconds before retrying
[task 2024-02-09T22:24:40.202Z] 22:24:40     INFO - retry: Calling run_command with args: [['/opt/worker/tasks/task_170751661357128/build/venv/bin/pip', 'install', '--timeout', '120', '-r', '/opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport/requirements.txt', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub/']], kwargs: {'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': re.compile('\\d+: error: '), 'level': 'error'}, {'regex': re.compile('\\d+: warning: '), 'level': 'warning'}, {'regex': re.compile('Downloading .* \\(.*\\): *([0-9]+%)? *[0-9\\.]+[kmKM]b'), 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': re.compile('Warning:.*Error: '), 'level': 'warning'}, {'regex': re.compile('package.*> Error:'), 'level': 'error'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': re.compile('raise \\w*Exception: '), 'level': 'critical'}, {'regex': re.compile('raise \\w*Error: '), 'level': 'critical'}], 'cwd': '/opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport', 'env': {'RUN_ID': '0', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.ipvB7MZsh7/Listeners', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin', '_': '/usr/local/bin/start-worker', 'PYTHON': '/usr/local/bin/python3', 'TESTS_BY_MANIFEST_URL': 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Sy-6go-0RJOO50S37LFKkg/artifacts/public/tests-by-manifest.json.gz', 'MOZ_NODE_PATH': '/usr/local/bin/node', 'TASKCLUSTER_ROOT_URL': 'https://firefox-ci-tc.services.mozilla.com', 'SHLVL': '1', 'HOME': '/Users/cltbld', 'EXTRA_MOZHARNESS_CONFIG': '{"installer_url": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TqjVJnZuTWiQV1VC3CF0IA/artifacts/public/build/target.dmg", "test_packages_url": "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TqjVJnZuTWiQV1VC3CF0IA/artifacts/public/build/target.test_packages.json"}', 'MOZ_AUTOMATION': '1', 'TASKCLUSTER_WORKER_LOCATION': '{"cloud":"standalone","host":"mdc1"}', 'PWD': '/opt/worker', 'XPC_SERVICE_NAME': '0', 'MOZHARNESS_TEST_PATHS': '{"web-platform-tests": ["/WebCryptoAPI/generateKey", "/_mozilla/editor", "/_mozilla/service-workers", "/audio-output", "/bluetooth/characteristic/writeValueWithoutResponse", "/bluetooth/service", "/clipboard-apis/detached-iframe", "/content-security-policy/connect-src", "/content-security-policy/navigation", "/content-security-policy/wasm-unsafe-eval", "/cookies/secure", "/css/CSS2/visufx", "/css/css-animations/parsing", "/css/css-color-adjust/animation", "/css/css-display/animations", "/css/css-fonts/math-script-level-and-math-style", "/css/css-images", "/css/css-logical/animations", "/css/css-page", "/css/css-scroll-anchoring", "/css/css-size-adjust", "/css/css-text-decor/animations", "/css/css-transforms", "/css/css-values/urls", "/css/mediaqueries", "/custom-elements/reactions/customized-builtins", "/dom/events/non-cancelable-when-passive", "/element-timing", "/feature-policy", "/fetch/content-type", "/fetch/range", "/generic-sensor", "/html/capability-delegation", "/html/infrastructure/common-dom-interfaces", "/html/scripting/the-script-element", "/html/semantics/the-link-element", "/html/webappapis/the-windoworworkerglobalscope-mixin", "/infrastructure/testdriver", "/keyboard-lock", "/mathml/presentation-markup/radicals", "/mediacapture-insertable-streams/legacy", "/mixed-content/gen/sharedworker-module-data.http-rp", "/navigation-api/navigation-activation", "/page-lifecycle", "/permissions", "/push-api", "/referrer-policy/gen/top.http-rp", "/screen-capture", "/serial/getPorts", "/shadow-dom/untriaged/styles", "/streams/piping", "/svg/geometry/parsing", "/svg/struct", "/uievents/constructors", "/upgrade-insecure-requests/gen/top.http-rp", "/wasm/jsapi/constructor", "/web-animations/animation-model/keyframe-effects", "/web-otp", "/webaudio/the-audio-api/the-dynamicscompressornode-interface", "/webidl", "/webrtc-extensions", "/websockets/multi-globals/url-parsing", "/webxr/ar-module", "/workers/interfaces/SharedWorkerGlobalScope"]}', 'MOZ_SCM_LEVEL': '3', 'SHELL': '/bin/bash', 'TMPDIR': '/var/folders/g7/vplxp93107b0ynpl9047dz78000014/T/', 'GECKO_HEAD_REV': '9c7562b7913101280cd24aef6e3839c02f297816', 'LC_ALL': 'en_US.UTF-8', 'LOGNAME': 'cltbld', 'MOZ_FETCHES_DIR': '/opt/worker/tasks/task_170751661357128/fetches', 'SCCACHE_DISABLE': '1', 'TASK_ID': 'eRldTU2LSTiLeo3bY6kzTQ', 'USER': 'cltbld', 'XPC_FLAGS': '0x0', 'MOZ_FETCHES': '[{"artifact": "public/build/fix-stacks.tar.zst", "extract": true, "task": "N_d0vLwWRL2Pw4SXbVzd4g"}, {"artifact": "public/build/minidump-stackwalk.tar.zst", "extract": true, "task": "YYqYnZtrTBy9VUhWOw7LxA"}]', 'LANG': 'en_US.UTF-8', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/mozilla-central', '__CF_USER_TEXT_ENCODING': '0x24:0x0:0x0', 'REQUIRE_GPU': '0'}, 'error_level': 'warning'}, attempt #5
[task 2024-02-09T22:24:40.202Z] 22:24:40     INFO - Running command: ['/opt/worker/tasks/task_170751661357128/build/venv/bin/pip', 'install', '--timeout', '120', '-r', '/opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport/requirements.txt', '--no-index', '--find-links', 'https://pypi.pub.build.mozilla.org/pub/'] in /opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport
[task 2024-02-09T22:24:40.202Z] 22:24:40     INFO - Copy/paste: /opt/worker/tasks/task_170751661357128/build/venv/bin/pip install --timeout 120 -r /opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport/requirements.txt --no-index --find-links https://pypi.pub.build.mozilla.org/pub/
[task 2024-02-09T22:24:40.202Z] 22:24:40     INFO - Using env: (same as previous command)
[task 2024-02-09T22:24:40.625Z] 22:24:40     INFO -  Looking in links: https://pypi.pub.build.mozilla.org/pub/, https://pypi.pub.build.mozilla.org/pub/
[task 2024-02-09T22:24:40.626Z] 22:24:40     INFO -  Requirement already satisfied: aioquic==0.9.21 in /opt/worker/tasks/task_170751661357128/build/venv/lib/python3.7/site-packages (from -r /opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport/requirements.txt (line 1)) (0.9.21)
[task 2024-02-09T22:24:40.632Z] 22:24:40     INFO -  Requirement already satisfied: certifi in /opt/worker/tasks/task_170751661357128/build/venv/lib/python3.7/site-packages (from aioquic==0.9.21->-r /opt/worker/tasks/task_170751661357128/build/tests/web-platform/tests/tools/webtransport/requirements.txt (line 1)) (2018.4.16)
[task 2024-02-09T22:24:41.217Z] 22:24:41     INFO -  Collecting pyopenssl>=22
[task 2024-02-09T22:24:41.223Z] 22:24:41     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/pyOpenSSL-23.2.0-py3-none-any.whl (59 kB)
[task 2024-02-09T22:24:41.333Z] 22:24:41     INFO -  Collecting pylsqpack<0.4.0,>=0.3.3
[task 2024-02-09T22:24:41.341Z] 22:24:41     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/pylsqpack-0.3.17-cp37-cp37m-macosx_10_9_x86_64.whl (161 kB)
[task 2024-02-09T22:24:41.458Z] 22:24:41     INFO -  Collecting cryptography!=40.0.0,!=40.0.1,<42,>=38.0.0
[task 2024-02-09T22:24:41.482Z] 22:24:41     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/cryptography-41.0.2-cp37-abi3-macosx_10_12_x86_64.whl (2.8 MB)
[task 2024-02-09T22:24:41.637Z] 22:24:41     INFO -  Collecting cffi>=1.12
[task 2024-02-09T22:24:41.646Z] 22:24:41     INFO -    Using cached https://pypi.pub.build.mozilla.org/pub/cffi-1.16.0.tar.gz (512 kB)
[task 2024-02-09T22:24:41.772Z] 22:24:41     INFO -    Installing build dependencies: started
[task 2024-02-09T22:24:42.826Z] 22:24:42     INFO -    Installing build dependencies: finished with status 'error'
[task 2024-02-09T22:24:42.833Z] 22:24:42     INFO -    error: subprocess-exited-with-error
[task 2024-02-09T22:24:42.833Z] 22:24:42     INFO -    × pip subprocess to install build dependencies did not run successfully.
[task 2024-02-09T22:24:42.833Z] 22:24:42     INFO -    │ exit code: 1
[task 2024-02-09T22:24:42.833Z] 22:24:42     INFO -    ╰─> [3 lines of output]
[task 2024-02-09T22:24:42.833Z] 22:24:42     INFO -        Looking in links: https://pypi.pub.build.mozilla.org/pub/, https://pypi.pub.build.mozilla.org/pub/, https://pypi.pub.build.mozilla.org/pub/
[task 2024-02-09T22:24:42.833Z] 22:24:42     INFO -        ERROR: Could not find a version that satisfies the requirement setuptools>=66.1 (from versions: rust-0.12.1, 4.0.1, 41.2.0, 43.0.0, 57.0.0, 59.2.0)
[task 2024-02-09T22:24:42.833Z] 22:24:42     INFO -        ERROR: No matching distribution found for setuptools>=66.1
[task 2024-02-09T22:24:42.834Z] 22:24:42     INFO -        [end of output]
[task 2024-02-09T22:24:42.834Z] 22:24:42     INFO -    note: This error originates from a subprocess, and is likely not a problem with pip.
[task 2024-02-09T22:24:42.835Z] 22:24:42     INFO -  error: subprocess-exited-with-error
[task 2024-02-09T22:24:42.835Z] 22:24:42     INFO -  × pip subprocess to install build dependencies did not run successfully.
[task 2024-02-09T22:24:42.835Z] 22:24:42     INFO -  │ exit code: 1
[task 2024-02-09T22:24:42.835Z] 22:24:42     INFO -  ╰─> See above for output.
[task 2024-02-09T22:24:42.835Z] 22:24:42     INFO -  note: This error originates from a subprocess, and is likely not a problem with pip.
[task 2024-02-09T22:24:42.924Z] 22:24:42     INFO - Return code: 1
[task 2024-02-09T22:24:42.924Z] 22:24:42    FATAL - Could not install python package: failed all attempts.
[task 2024-02-09T22:24:42.924Z] 22:24:42    FATAL - Running post_fatal callback...
[task 2024-02-09T22:24:42.924Z] 22:24:42    FATAL - Exiting -1
[task 2024-02-09T22:24:42.924Z] 22:24:42     INFO - Running post-action listener: _resource_record_post_action
[task 2024-02-09T22:24:42.924Z] 22:24:42     INFO - Running post-action listener: _start_resource_monitoring
[task 2024-02-09T22:24:42.925Z] 22:24:42     INFO - Starting resource monitoring.
[task 2024-02-09T22:24:42.944Z] 22:24:42     INFO - [mozharness: 2024-02-09 22:24:42.943845Z] Finished create-virtualenv step (failed)
[task 2024-02-09T22:24:42.944Z] 22:24:42     INFO - Running post-run listener: _resource_record_post_run
[task 2024-02-09T22:24:43.010Z] 22:24:43     INFO - Total resource usage - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-02-09T22:24:43.010Z] 22:24:43     INFO - TinderboxPrint: I/O read bytes / time<br/>0 / 0
[task 2024-02-09T22:24:43.010Z] 22:24:43     INFO - TinderboxPrint: I/O write bytes / time<br/>0 / 0
[task 2024-02-09T22:24:43.010Z] 22:24:43     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[taskcluster 2024-02-09T22:24:43.039Z]    Exit Code: 255
[taskcluster 2024-02-09T22:24:43.039Z]    User Time: 1m32.624826s
[taskcluster 2024-02-09T22:24:43.039Z]  Kernel Time: 41.924327s
[taskcluster 2024-02-09T22:24:43.039Z]    Wall Time: 14m26.411286s
[taskcluster 2024-02-09T22:24:43.039Z]       Result: FAILED
[taskcluster 2024-02-09T22:24:43.039Z] === Task Finished ===
[taskcluster 2024-02-09T22:24:43.039Z] Task Duration: 14m26.440197s
[taskcluster 2024-02-09T22:24:44.152Z] Uploading artifact public/logs/localconfig.json from file /opt/worker/tasks/task_170751661357128/logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2024-05-09T21:46:28.018Z
[taskcluster 2024-02-09T22:24:44.635Z] Uploading artifact public/test_info/profile_resource-usage.json from file /opt/worker/tasks/task_170751661357128/build/blobber_upload_dir/profile_resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-05-09T21:46:28.018Z
[taskcluster 2024-02-09T22:24:44.896Z] Uploading artifact public/test_info/resource-usage.json from file /opt/worker/tasks/task_170751661357128/build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2024-05-09T21:46:28.018Z
[taskcluster 2024-02-09T22:24:45.522Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-05-09T21:46:28.018Z
[taskcluster:error] exit status 255

Comment 1

[Cristian Tuns]

Hi, can you please take a look at this? it is happening on OS X 10.15 WebRender on central and autoland.
https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&searchStr=os%2Cx%2C10.15%2Cwebrender%2Cdebug%2C&revision=9c7562b7913101280cd24aef6e3839c02f297816&selectedTaskRun=Fy8dsTjTSW6wh_3h6AygfA.0

Comment 2

[[e-mail] (:bhearsum)]

I doubt this is an issue with the image. This is happening when trying to install https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/webtransport/requirements.txt, which doesn't fully pin its requirements. Specifically, we're unable to find a new enough setuptools:

[task 2024-02-09T22:15:37.187Z] 22:15:37     INFO -        ERROR: Could not find a version that satisfies the requirement setuptools>=66.1 (from versions: rust-0.12.1, 4.0.1, 41.2.0, 43.0.0, 57.0.0, 59.2.0)
[task 2024-02-09T22:15:37.187Z] 22:15:37     INFO -        ERROR: No matching distribution found for setuptools>=66.1

We should do a couple of things here:

• PIn these requirements properly, by moving the existing file to requirements.in, and then running pip-compile -r requirements.in --generate-hashes to generate a fully pinned requirements file.
• Upload any necessary new packages to our internal pypi mirror

(We should also do this for https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wpt, https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner, and any other requirements files in web-platform to guard against future such errors.)

I'm not sure whether or not these are files that are imported as part of wpt sync, but even if they are we need to find a way to make the requirements stable IMO.

Comment 3

[Iulian Moraru]

This started to affect Linux on esr115 as it can be seen here.

Hi Geoff! Can you please take a look at this? Ryan mentioned that you might have helped with this in the past. - "if we can find someone with permissions to do so, the "Upload any necessary new packages to our internal pypi mirror" option seems like the fastest way out of this"
Thank you!

Comment 4

[Cosmin Sabou [:CosminS]]

Adding Joel to ni list, maybe he knows something about this CI issue. Thanks.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

hmm, this looks related to the work I was doing yesterday on getting python 3.11 working on osx 10.15 and windows11. The error is different, but only in version numbers. I think if we got setuptools available we might get past this.

I think most likely what happened is we needed a package > version, and now we found the newest version and install it which doesn't necessarily work with other packages/python versions.

To :bhearsum's point, we should pin requirements for wpt.

to fix this, we should install some packages.

here is a wheel for the latest setuptools (69.0.3):
https://files.pythonhosted.org/packages/55/3a/5121b58b578a598b269537e09a316ad2a94fdd561a2c6eb75cd68578cc6b/setuptools-69.0.3-py3-none-any.whl

rcurran are you around this weekend and can you upload ^

Comment 6

[Geoff Brown [:gbrown]]

I uploaded setuptools==69.0.3 and retriggered some tests. Sometimes there are multiple problems and some iterations are needed -- not sure we are in the clear yet. I'll check back later today.

Comment 7

[Geoff Brown [:gbrown]]

[task 2024-02-10T20:29:28.364Z] 20:29:28     INFO -        Collecting setuptools>=66.1
[task 2024-02-10T20:29:28.364Z] 20:29:28     INFO -          Downloading https://pypi.pub.build.mozilla.org/pub/setuptools-69.0.3-py3-none-any.whl (819 kB)
[task 2024-02-10T20:29:28.364Z] 20:29:28     INFO -             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.5/819.5 kB 24.3 MB/s eta 0:00:00
[task 2024-02-10T20:29:28.364Z] 20:29:28     INFO -        ERROR: Package 'setuptools' requires a different Python: 3.7.9 not in '>=3.8'

Trying again with setuptools==68.0.0, which should be okay with python 3.7.

Comment 8

[Cosmin Sabou [:CosminS]]

After comment 6 I've retriggered some jobs on autoland and turned out green but the lines from comment 7 are there so maybe they don't turn the jobs orange?
https://treeherder.mozilla.org/logviewer?job_id=446822014&repo=autoland&lineNumber=1100

https://treeherder.mozilla.org/logviewer?job_id=446822015&repo=autoland&lineNumber=989

Comment 9

[Cosmin Sabou [:CosminS]]

Retriggers are turning green.

Comment 10

[Geoff Brown [:gbrown]]

lgtm. Thanks for the retriggers.

Comment 11

[Cosmin Sabou [:CosminS]]

No more failures after those changes, considering this fixed, leaving the ni so that James could have a look over the points raised.

Comment 12

[Intermittent Failures Robot]

727 failures in 120 pushes (6.058 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 217
• try: 93
• mozilla-beta: 88
• mozilla-central: 69
• mozilla-esr115: 260

Platform and build breakdown:

• macosx1015-64-qr: 364
  • debug: 275
  • opt: 89
• macosx1015-64-shippable-qr: 88
  • opt: 88
• macosx1015-64-devedition-qr: 50
  • opt: 50
• linux1804-64-qr: 80
  • debug: 80
• linux1804-64-tsan-qr: 66
  • opt: 66
• windows11-32-2009-qr: 2
  • debug: 2
• linux1804-64-asan-qr: 40
  • opt: 40
• linux1804-64-shippable-qr: 26
  • opt: 26
• linux1804-32-shippable-qr: 10
  • opt: 10
• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1879665&startday=2024-02-10&endday=2024-02-10&tree=all

Comment 13

[Serban Stanca [:SerbanS]]

Hi Geoff! Could you also please take a look at this? It's perma failing even before, where it was green. It's happening only on Windows 7 WebRender Shippable and on mozilla-esr115. Could they be related?

Thank you!

Comment 14

[Intermittent Failures Robot]

935 failures in 4009 pushes (0.233 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 199
• autoland: 291
• try: 93
• mozilla-beta: 88
• mozilla-esr115: 264

Platform and build breakdown:

• macosx1015-64-qr: 554
  • debug: 435
  • opt: 119
• macosx1015-64-shippable-qr: 102
  • opt: 102
• macosx1015-64-devedition-qr: 50
  • opt: 50
• linux1804-64-qr: 80
  • debug: 80
• linux1804-64-tsan-qr: 66
  • opt: 66
• windows11-32-2009-qr: 2
  • debug: 2
• linux1804-64-asan-qr: 40
  • opt: 40
• linux1804-64-shippable-qr: 26
  • opt: 26
• linux1804-32-shippable-qr: 10
  • opt: 10
• windows7-32-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1879665&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 15

[Geoff Brown [:gbrown]]

(In reply to Serban Stanca [:SerbanS] from comment #13)
That looks a little different. Following up in bug 1879915.