1964540 - Perma esr115 - ERROR: could not install deps | Could not fetch URL https://pypi.python.org/simple/coverage/: There was a problem confirming the ssl certificate

Status: RESOLVED DUPLICATE of bug 1943670

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Cosmin Sabou [:CosminS]]

Failure log: https://treeherder.mozilla.org/logviewer?job_id=506716451&repo=mozilla-esr115&lineNumber=234
Treeherder link

[vcs 2025-05-05T22:17:29.945Z] 341765 files updated, 0 files merged, 0 files removed, 0 files unresolved
[vcs 2025-05-05T22:17:30.056Z] updated to 77e052aeb1a613fc265367f073ebbc6790828844
[vcs 2025-05-05T22:17:30.057Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 115.90365529060364}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "update", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 76.53347301483154}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 193.38463044166565}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 193.38463044166565}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 193.38463044166565}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_emptywdir", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 193.38463044166565}]}
[vcs 2025-05-05T22:17:30.415Z] TinderboxPrint:<a href=https://hg.mozilla.org/releases/mozilla-esr115/rev/77e052aeb1a613fc265367f073ebbc6790828844 title='Built from mozilla-esr115 revision 77e052aeb1a613fc265367f073ebbc6790828844'>77e052aeb1a613fc265367f073ebbc6790828844</a>
[setup 2025-05-05T22:17:30.415Z] GECKO_PATH is /builds/worker/checkouts/gecko
[task 2025-05-05T22:17:30.415Z] executing ['bash', '-cx', '/usr/local/bin/tox -e py39-hg5.2\n']in /builds/worker/checkouts/gecko/testing/mozharness
[task 2025-05-05T22:17:30.420Z] bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[task 2025-05-05T22:17:30.420Z] + /usr/local/bin/tox -e py39-hg5.2
[task 2025-05-05T22:17:30.520Z] GLOB sdist-make: /builds/worker/checkouts/gecko/testing/mozharness/setup.py
[task 2025-05-05T22:17:30.705Z] py39-hg5.2 create: /builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2
[task 2025-05-05T22:17:32.533Z] py39-hg5.2 installdeps: coverage, distro, nose, rednose, PyYAML==6.0, /builds/worker/checkouts/gecko/testing/mozbase/mozlog, mercurial==5.2.1
[task 2025-05-05T22:17:32.983Z] ERROR: invocation failed (exit code 1), logfile: /builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2/log/py39-hg5.2-1.log
[task 2025-05-05T22:17:32.984Z] ERROR: actionid: py39-hg5.2
[task 2025-05-05T22:17:32.984Z] msg: getenv
[task 2025-05-05T22:17:32.984Z] cmdargs: ['/builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2/bin/pip', 'install', 'coverage', 'distro', 'nose', 'rednose', 'PyYAML==6.0', local('/builds/worker/checkouts/gecko/testing/mozbase/mozlog'), 'mercurial==5.2.1']
[task 2025-05-05T22:17:32.984Z] env: {'SHELL': '/bin/bash', 'GECKO_HEAD_REV': '77e052aeb1a613fc265367f073ebbc6790828844', 'HOSTNAME': 'taskcluster-worker', 'MOZ_AUTOMATION': '1', 'TASKCLUSTER_ROOT_URL': 'https://firefox-ci-tc.services.mozilla.com', 'PWD': '/builds/worker/checkouts/gecko/testing/mozharness', 'LOGNAME': 'worker', 'GECKO_PATH': '/builds/worker/checkouts/gecko', 'TASKCLUSTER_PORT_80_TCP_PORT': '80', 'HOME': '/builds/worker', 'TASKCLUSTER_WORKER_GROUP': 'us-central1-a', 'LANG': 'en_US.UTF-8', 'TASKCLUSTER_NAME': '/happy_visvesvaraya/taskcluster', 'TASKCLUSTER_INSTANCE_TYPE': 'projects/887720501152/machineTypes/n2-standard-4', 'GECKO_BASE_REPOSITORY': 'https://hg.mozilla.org/mozilla-unified', 'TASKCLUSTER_PORT': 'tcp://172.17.0.3:80', 'TASKCLUSTER_WORKER_LOCATION': '{"cloud":"google","region":"us-central1","zone":"us-central1-a"}', 'TERM': 'xterm', 'TASKCLUSTER_PUBLIC_IP': '35.202.113.116', 'HG_STORE_PATH': '/builds/worker/checkouts/hg-store', 'USER': 'worker', 'TASKCLUSTER_PORT_80_TCP_PROTO': 'tcp', 'MOZ_SCM_LEVEL': '3', 'SHLVL': '0', 'TASKCLUSTER_PROXY_URL': 'http://taskcluster', 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/releases/mozilla-esr115', 'SCCACHE_DISABLE': '1', 'LC_ALL': 'en_US.UTF-8', 'PATH': '/builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'RUN_ID': '0', 'TASK_ID': 'dlL5Q2Z6QgexusgpfNiioA', 'TASKCLUSTER_WORKER_TYPE': 't-linux-xlarge-source-gcp', 'TASKCLUSTER_PORT_80_TCP_ADDR': '172.17.0.3', 'DEBIAN_FRONTEND': 'noninteractive', 'TASKCLUSTER_PORT_80_TCP': 'tcp://172.17.0.3:80', '_': '/usr/local/bin/tox', 'HGRCPATH': '/builds/worker/checkouts/gecko/testing/mozharness/test/hgrc', 'PYTHONPATH': '$PYTHONPATH:/builds/worker/checkouts/gecko/testing/mozharness/../mozbase//manifestparser:/builds/worker/checkouts/gecko/testing/mozharness/../mozbase//mozfile:/builds/worker/checkouts/gecko/testing/mozharness/../mozbase//mozinfo:/builds/worker/checkouts/gecko/testing/mozharness/../mozbase//mozprocess', 'PYTHONHASHSEED': '3490757411', 'VIRTUAL_ENV': '/builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2'}
[task 2025-05-05T22:17:32.984Z] 
[task 2025-05-05T22:17:32.984Z] Processing /builds/worker/checkouts/gecko/testing/mozbase/mozlog
[task 2025-05-05T22:17:32.984Z] Collecting coverage
[task 2025-05-05T22:17:32.984Z]   Could not fetch URL https://pypi.python.org/simple/coverage/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123) - skipping
[task 2025-05-05T22:17:32.984Z]   Could not find a version that satisfies the requirement coverage (from versions: )
[task 2025-05-05T22:17:32.984Z] No matching distribution found for coverage
[task 2025-05-05T22:17:32.984Z] 
[task 2025-05-05T22:17:32.984Z] ERROR: could not install deps [coverage, distro, nose, rednose, PyYAML==6.0, /builds/worker/checkouts/gecko/testing/mozharness/../mozbase/mozlog, mercurial==5.2.1]; v = InvocationError('/builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2/bin/pip install coverage distro nose rednose PyYAML==6.0 /builds/worker/checkouts/gecko/testing/mozbase/mozlog mercurial==5.2.1 (see /builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2/log/py39-hg5.2-1.log)', 1)
[task 2025-05-05T22:17:32.985Z] ___________________________________ summary ____________________________________
[task 2025-05-05T22:17:32.985Z] ERROR:   py39-hg5.2: could not install deps [coverage, distro, nose, rednose, PyYAML==6.0, /builds/worker/checkouts/gecko/testing/mozharness/../mozbase/mozlog, mercurial==5.2.1]; v = InvocationError('/builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2/bin/pip install coverage distro nose rednose PyYAML==6.0 /builds/worker/checkouts/gecko/testing/mozbase/mozlog mercurial==5.2.1 (see /builds/worker/checkouts/gecko/testing/mozharness/.tox/py39-hg5.2/log/py39-hg5.2-1.log)', 1)
[taskcluster 2025-05-05 22:17:35.114Z] === Task Finished ===
[taskcluster 2025-05-05 22:17:35.114Z] Unsuccessful task run with exit code: 1 completed in 221.388 seconds

Comment 1

[[e-mail] (:bhearsum)]

AFAICT this has been broken for awhile, and these tests haven't run until I pushed a recent change. I think the problem is that the certifi that pip bundles (or maybe the one we vendor...) is too old. Hopefully upgrading one or the other will not cause other issues...

Comment 2

[[e-mail] (:bhearsum)]

Okay...so somehow the virtualenv created when running tox ends up with pip 9, despite it not being present on the docker image...that would almost certainly explain this. I'm not sure where pip 9 is coming from yet though...

Comment 3

[[e-mail] (:bhearsum)]

I suspect backporting https://bugzilla.mozilla.org/show_bug.cgi?id=1943670 will fix this; I'm going to look at that.

Comment 4

[[e-mail] (:bhearsum)]

This will be fixed by the backport happening in https://bugzilla.mozilla.org/show_bug.cgi?id=1943670