Closed Bug 1650762 Opened 4 years ago Closed 4 years ago

failure to resolve SSL at files.pythonhosted.org while setting up jobs with pip install for python packages

Categories

(Infrastructure & Operations :: RelOps: Windows OS, task)

Product:
Infrastructure & Operations
Component:
RelOps: Windows OS
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jmaher, Assigned: jmaher)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Bug 1650762 - ensure we have ssl certs on windows when mozharness is doing pip installs. r=aki,rail
47 bytes, text/x-phabricator-request
Details | Review

in this try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9c2571e41aefa518ad62e8cd4a0916b2eedbe543

I am trying to run unittests that normally run at AWS on datacenter moonshots. This is for the Windows 10 tests specifically.

in the mochitest failures, I see:
https://firefoxci.taskcluster-artifacts.net/Sri74JqxSFGNSrujMmCx1w/0/public/logs/live_backing.log

[task 2020-07-01T15:43:32.595Z] 15:43:32 INFO - Copy/paste: C:\Users\task_1593617634\build\venv\Scripts\pip install --no-deps --timeout 120 -r C:\Users\task_1593617634\build\tests\mochitest\websocketprocessbridge\websocketprocessbridge_requirements.txt --no-index --find-links https://pypi.pub.build.mozilla.org/pub
[task 2020-07-01T15:43:32.595Z] 15:43:32 INFO - Using env: (same as previous command)
[task 2020-07-01T15:43:32.852Z] 15:43:32 INFO - Ignoring indexes: https://pypi.python.org/simple
[task 2020-07-01T15:43:32.861Z] 15:43:32 INFO - Requirement already satisfied (use --upgrade to upgrade): six in c:\users\task_1593617634\build\venv\lib\site-packages (from -r C:\Users\task_1593617634\build\tests\mochitest\websocketprocessbridge\websocketprocessbridge_requirements.txt (line 2))
[task 2020-07-01T15:43:32.874Z] 15:43:32 INFO - Collecting twisted==10.2.0 (from -r C:\Users\task_1593617634\build\tests\mochitest\websocketprocessbridge\websocketprocessbridge_requirements.txt (line 4))
[task 2020-07-01T15:43:34.499Z] 15:43:34 INFO - Downloading https://pypi.pub.build.mozilla.org/pub/Twisted-10.2.0.tar.gz (1.9MB)
[task 2020-07-01T15:43:36.072Z] 15:43:36 INFO - Collecting txws==0.9.1 (from -r C:\Users\task_1593617634\build\tests\mochitest\websocketprocessbridge\websocketprocessbridge_requirements.txt (line 7))
[task 2020-07-01T15:43:36.888Z] 15:43:36 INFO - Downloading https://pypi.pub.build.mozilla.org/pub/txWS-0.9.1.tar.gz
[task 2020-07-01T15:43:37.541Z] 15:43:37 INFO - Complete output from command python setup.py egg_info:
[task 2020-07-01T15:43:37.542Z] 15:43:37 ERROR - Traceback (most recent call last):
[task 2020-07-01T15:43:37.542Z] 15:43:37 INFO - File "<string>", line 1, in <module>
[task 2020-07-01T15:43:37.542Z] 15:43:37 INFO - File "c:\users\task_1593617634\appdata\local\temp\pip-build-wirubp\txws\setup.py", line 15, in <module>
[task 2020-07-01T15:43:37.542Z] 15:43:37 INFO - url="http://github.com/MostAwesomeDude/txWS",
[task 2020-07-01T15:43:37.542Z] 15:43:37 INFO - File "c:\mozilla-build\python\Lib\distutils\core.py", line 111, in setup
[task 2020-07-01T15:43:37.542Z] 15:43:37 INFO - setup_distribution = dist = klass(attrs)
[task 2020-07-01T15:43:37.542Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\dist.py", line 348, in init
[task 2020-07-01T15:43:37.542Z] 15:43:37 INFO - self.fetch_build_eggs(attrs['setup_requires'])
[task 2020-07-01T15:43:37.543Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\dist.py", line 394, in fetch_build_eggs
[task 2020-07-01T15:43:37.543Z] 15:43:37 INFO - replace_conflicting=True,
[task 2020-07-01T15:43:37.543Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\pkg_resources_init.py", line 826, in resolve
[task 2020-07-01T15:43:37.543Z] 15:43:37 INFO - dist = best[req.key] = env.best_match(req, ws, installer)
[task 2020-07-01T15:43:37.543Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\pkg_resources_init_.py", line 1098, in best_match
[task 2020-07-01T15:43:37.543Z] 15:43:37 INFO - return self.obtain(req, installer)
[task 2020-07-01T15:43:37.543Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\pkg_resources_init_.py", line 1110, in obtain
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - return installer(requirement)
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\dist.py", line 461, in fetch_build_egg
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - return cmd.easy_install(req)
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\command\easy_install.py", line 652, in easy_install
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - not self.always_copy, self.local_index
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\package_index.py", line 631, in fetch_distribution
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - dist = find(requirement)
[task 2020-07-01T15:43:37.544Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\package_index.py", line 612, in find
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - dist.download_location = self.download(dist.location, tmpdir)
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\package_index.py", line 555, in download
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - found = self._download_url(scheme.group(1), spec, tmpdir)
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\package_index.py", line 800, in _download_url
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - return self._attempt_download(url, filename)
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\package_index.py", line 806, in _attempt_download
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - headers = self._download_to(url, filename)
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\package_index.py", line 705, in _download_to
[task 2020-07-01T15:43:37.545Z] 15:43:37 INFO - fp = self.open_url(strip_fragment(url))
[task 2020-07-01T15:43:37.546Z] 15:43:37 INFO - File "c:\users\task_1593617634\build\venv\lib\site-packages\setuptools\package_index.py", line 756, in open_url
[task 2020-07-01T15:43:37.546Z] 15:43:37 INFO - % (url, v.reason))
[task 2020-07-01T15:43:37.546Z] 15:43:37 INFO - distutils.errors.DistutilsError: Download error for https://files.pythonhosted.org/packages/c5/cc/33162c0a7b28a4d8c83da07bc2b12cee58c120b4a9e8bba31c41c8d35a16/vcversioner-2.16.0.0.tar.gz#sha256=dae60c17a479781f44a4010701833f1829140b1eeccd258762a74974aa06e19b: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
[task 2020-07-01T15:43:37.546Z] 15:43:37 INFO - ----------------------------------------
[task 2020-07-01T15:43:37.663Z] 15:43:37 INFO - Command "python setup.py egg_info" failed with error code 1 in c:\users\task_1593617634\appdata\local\temp\pip-build-wirubp\txws
[task 2020-07-01T15:43:37.704Z] 15:43:37 WARNING - Return code: 1

We have vcversioner on our internal pypi:
https://pypi.pub.build.mozilla.org/pub/vcversioner-2.16.0.0.tar.gz

But for some reason it is failing with:
Download error for https://files.pythonhosted.org/packages/c5/cc/33162c0a7b28a4d8c83da07bc2b12cee58c120b4a9e8bba31c41c8d35a16/vcversioner-2.16.0.0.tar.gz#sha256=dae60c17a479781f44a4010701833f1829140b1eeccd258762a74974aa06e19b: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)

I wonder if the AWS VM installation is different that moonshot installation, specifically for pip.ini to point to pypi.pub.build.mozilla.org

:fubar, can you help me find the right person who could answer this?

Flags: needinfo?(klibby)

That's odd. Mark, can you look when you get a chance?

Flags: needinfo?(klibby) → needinfo?(mcornmesser)

The pip.ini file is identical for both AWS and the datacenters. Both have "trusted-host = pypi.pub.build.mozilla.org". I will take a look into this later this week.

I took a bit of a deeper look into this. There was no obvious difference between Python install in AWS and the datacenter. Also I had no issue doing a wget for the file that was mentioned in the download error.

$ wget https://files.pythonhosted.org/packages/c5/cc/33162c0a7b28a4d8c83da07bc2b12cee58c120b4a9e8bba31c41c8d35a16/vcve
rsioner-2.16.0.0.tar.gz#sha256=dae60c17a479781f44a4010701833f1829140b1eeccd258762a74974aa06e19b
--2020-07-14 22:39:51-- https://files.pythonhosted.org/packages/c5/cc/33162c0a7b28a4d8c83da07bc2b12cee58c120b4a9e8bba31c41c8d35a16/vcversioner-2.16.0.0.tar.gz
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.41.63
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.41.63|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9024 (8.8K) [application/octet-stream]
Saving to: 'vcversioner-2.16.0.0.tar.gz'

vcversioner-2.16.0.0.tar.gz 100%[=================================================>] 8.81K --.-KB/s in 0.003s

2020-07-14 22:39:51 (2.70 MB/s) - 'vcversioner-2.16.0.0.tar.gz' saved [9024/9024]

Form the worker configuration side i am not seeing why there is a redirect to https://files.pythonhosted.org ... nor a reason for the ssl error. There is a difference in the generic-worker version, AWS 16.5.0 and datacenter 16.2.0. That might have an effect on the creation of the virtual environment, but that is just a guess.

Flags: needinfo?(mcornmesser)

appears that the certs need to be primed prior to needing them (https://bugs.python.org/issue36011). Working on testing that out now.

this seems to work and we can patch up mozharness to support this without too much controversy. This might future proof our work in future migrations.

ensure we have ssl certs on windows when mozharness is doing pip installs.

Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/df03b4231419
ensure we have ssl certs on windows when mozharness is doing pip installs. r=aki,rail
Depends on: 1655837
Blocks: 1655837
No longer depends on: 1655837

https://hg.mozilla.org/mozilla-central/rev/df03b4231419

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
See Also: → 965027
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: