1446296 - Builds install python packages from pypi.python.org

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Nick Thomas [:nthomas] (UTC+12)]

tl;dr: Windows builds on ESR52 use https://pypi.python.org to install python packages, and we're vulnerable to bustage as they transition to a new system which has a different SSL setup. We should use our internal pypi anyway!

Long version:
Earlier today Windows builds on esr52 were broken (eg https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-esr52-win64/1521156813/mozilla-esr52-win64-bm73-build1-build39.txt.gz). While mozharness attempted to install requests there were several retries for:

16:55:40     INFO -  Exception:
16:55:40    ERROR -  Traceback (most recent call last):
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\basecommand.py", line 215, in main
16:55:40     INFO -      status = self.run(options, args)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\commands\install.py", line 310, in run
16:55:40     INFO -      wb.build(autobuilding=True)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\wheel.py", line 750, in build
16:55:40     INFO -      self.requirement_set.prepare_files(self.finder)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\req\req_set.py", line 370, in prepare_files
16:55:40     INFO -      ignore_dependencies=self.ignore_dependencies))
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\req\req_set.py", line 587, in _prepare_file
16:55:40     INFO -      session=self.session, hashes=hashes)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\download.py", line 810, in unpack_url
16:55:40     INFO -      hashes=hashes
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\download.py", line 649, in unpack_http_url
16:55:40     INFO -      hashes)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\download.py", line 842, in _download_http_url
16:55:40     INFO -      stream=True,
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\_vendor\requests\sessions.py", line 487, in get
16:55:40     INFO -      return self.request('GET', url, **kwargs)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\download.py", line 378, in request
16:55:40     INFO -      return super(PipSession, self).request(method, url, *args, **kwargs)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\_vendor\requests\sessions.py", line 475, in request
16:55:40     INFO -      resp = self.send(prep, **send_kwargs)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\_vendor\requests\sessions.py", line 585, in send
16:55:40     INFO -      r = adapter.send(request, **kwargs)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 46, in send
16:55:40     INFO -      resp = super(CacheControlAdapter, self).send(request, **kw)
16:55:40     INFO -    File "c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\lib\site-packages\pip\_vendor\requests\adapters.py", line 477, in send
16:55:40     INFO -      raise SSLError(e, request=request)
16:55:40     INFO -  SSLError: [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
16:55:40     INFO -  You are using pip version 8.1.2, however version 9.0.1 is available.
16:55:40     INFO -  You should consider upgrading via the 'python -m pip install --upgrade pip' command.
16:55:40  WARNING - Return code: 2
16:55:40    FATAL - Could not install python package: c:\builds\moz2_slave\m-esr52-w64-000000000000000000\build\venv\Scripts\pip install --timeout 120 requests==2.8.1 failed after 5 tries!

The key line is the SSL error, but we were not futzing with out internal pypi server. To investigate further I did a try push with 'pip install -v' and 'pip freeze --all' to verify where we install from, versions of setuptools etc. The log for that is at https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-cc5d5e98bb717a5a5bda83df54bd9727fa08d55a/try-win32/try-win32-bm78-try1-build1.txt.gz, heres's the important bit:

18:00:46     INFO -  Collecting requests==2.8.1
18:00:46     INFO -    1 location(s) to search for versions of requests:
18:00:46     INFO -    * https://pypi.python.org/simple/requests/
...
18:00:47     INFO -    Downloading from URL https://pypi.python.org/packages/c0/0f/a911a44c89ba01b23d8fe3defbdfca1e962de6f11a11da32658902cdc2a4/requests-2.8.1-py2.py3-none-any.whl#md5=46f1d621daa3ab38958a42f51478b1ee (from https://pypi.python.org/simple/requests/)
18:00:47     INFO -    Updating cache with response from "https://pypi.python.org/packages/c0/0f/a911a44c89ba01b23d8fe3defbdfca1e962de6f11a11da32658902cdc2a4/requests-2.8.1-py2.py3-none-any.whl"
18:00:47     INFO -    Caching due to etag
18:00:47     INFO -  Installing collected packages: requests
18:00:48     INFO -  Successfully installed requests-2.8.1

OK, so pypi.python.org and not pypi.pvt.build.m.o. 

The problem resolved itself spontaneously. We think this was from Pypi doing load testing of their new hosting platform - https://wiki.python.org/psf/WarehouseRoadmap. eg https://status.python.org/incidents/0gmdf90kkt8n:

A subset of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

I suspect they are disabling old SSL protocols, similar to Github. Anyway, we should not be hitting pypi.python.org.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

mozharness's install_module() [https://dxr.mozilla.org/mozilla-esr52/source/testing/mozharness/mozharness/base/python.py#195] already has support for find_links and pip_index, just need to put it the right config. Test push which adds config: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d5c5d1a3098c4565cb551d35d2ba12d6664c768

Comment 2

[Chris AtLee [:catlee]]

Looks like you got green builds there.

Is there a reason this shouldn't be the default for all branches?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Turns out I modified the wrong config to test on try, so https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ac77a7747867afb1ca2f63626356d6fe4b6a591 is better (still green). 

Agree that this needs to be all branches/all platforms for builds. It probably shows up as bustage on ESR52 windows builds due to the python+ssl available there.

Comment 4

[Chris AtLee [:catlee]]

Looks good.

I think we should get rid of the -vvv for landing, it's really really spammy.

Comment 5

[Kendall Libby [:fubar] (he/him)]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #0)
> tl;dr: Windows builds on ESR52 use https://pypi.python.org to install python
> packages, and we're vulnerable to bustage as they transition to a new system
> which has a different SSL setup. We should use our internal pypi anyway!
> 
> [snip]
>
> The problem resolved itself spontaneously. We think this was from Pypi doing
> load testing of their new hosting platform -
> https://wiki.python.org/psf/WarehouseRoadmap. eg
> https://status.python.org/incidents/0gmdf90kkt8n:

https://status.python.org/incidents/ckj1v2ph3fvr

"In preparation for our CDN provider deprecating TLSv1.0 and TLSv1.1 protocols, we have begun rolling brownouts for these protocols for the first ten (10) minutes of each hour."

Comment 6

[Chris AtLee [:catlee]]

Some tests too:
https://treeherder.mozilla.org/logviewer.html#?job_id=169664700&repo=mozilla-inbound&lineNumber=1540

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko !esr52] Use releng_base_{firefox,android_64_builds}.py

Try push for similar patch @ https://treeherder.mozilla.org/#/jobs?repo=try&revision=d7595a29d487c7a8c2e2280ec8a10e24023c4043. I removed pypi.pvt.build.m.o because it fails DNS resolution from TC workers and is removed.

Seems to fix up all the desktop and android builds I checked.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko esr52] Use branch_specifics.py

This one is for esr52 only because it doesn't have an all of firefox config, so it's more convenient to use the branch specific config.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #8)

Try push for this at https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ac77a7747867afb1ca2f63626356d6fe4b6a591

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 8961635 [details] [diff] [review]
[gecko !esr52] Use releng_base_{firefox,android_64_builds}.py

The windows error on your try push seems odd, but also looks unrelated.

Comment 11

[Aki Sasaki (not active)]

Thanks!

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fef8790f544e
builds should use the internal pypi mirror, r=aki

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8961636 [details] [diff] [review]
[gecko esr52] Use branch_specifics.py

https://hg.mozilla.org/releases/mozilla-esr52/rev/fe2396d37fbd0232f5dcbe4c75ff273da3f44654

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

https://hg.mozilla.org/releases/mozilla-beta/rev/090520707f4da8ebdca18bd71ade9306851b1677
https://hg.mozilla.org/releases/mozilla-release/rev/ebd585545b53241a489bbe5171eecb119f3fe1b0

Comment 16

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/fef8790f544e

Comment 17

[Intermittent Failures Robot]

1 failures in 822 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1446296&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

I think that's a mis-star and bug 1448113 would have been a better choice.

Comment 19

[Intermittent Failures Robot]

2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1446296&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Hitting issues with installing requests during win32 l10n for ESR 52.7.4, due to branch_specifics.py only covering the build case.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [gecko esr52] Fix win l10n for releases

Mac and Linux jobs use our pvt pypi so just need to fix windows.

Comment 22

[Aki Sasaki (not active)]

Comment on attachment 8971749 [details] [diff] [review]
[gecko esr52] Fix win l10n for releases

We may want to add virtualenv_modules: ["requests"] while we're here?

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8971749 [details] [diff] [review]
[gecko esr52] Fix win l10n for releases

After discussion on IRC we added slugid to virutalenv_modules to avoid setuptools trying to install slugid from the main pypi.

https://hg.mozilla.org/releases/mozilla-esr52/rev/736c3ee75a56f8d0f2af81716f47ffd01df5d7e2 on FIREFOX_52_7_4esr_RELBRANCH
https://hg.mozilla.org/releases/mozilla-esr52/rev/146a2481508f585af656b257161b48296f45b7a5 on default

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Should have fixed configs/single_locale/win64.py too:

https://hg.mozilla.org/releases/mozilla-esr52/rev/aeaa3ce2f912210df7b4303062954cb6b2244d5c (relbranch)
https://hg.mozilla.org/releases/mozilla-esr52/rev/c4c625aa2509338ac6f1a587862b859a42f9338c (default)