Closed Bug 1388816 Opened 8 years ago Closed 7 years ago

Update pip on OSX

Categories

(Infrastructure & Operations :: RelOps: Puppet, task)

Product:
Infrastructure & Operations
Component:
RelOps: Puppet
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: fubar, Assigned: dragrom)

References

Details

Attachments

(1 file, 2 obsolete files)

Bug_1388816_Update_pip_on_OSX.patch
10.71 KB, patch
dividehex
: review-
Details | Diff | Splinter Review
From bug 1387970#c11: >> Linux docker images are using pip 9.x, while the new Windows TC workers have >> pip 8.1.1 installed. >> >> Wander, why are we so much behind on a recent version of pip on the OS X >> workers? Can we get this updated easily? > > pip version is deployed by puppet, I guess. The migration to TC worker didn't > touch any of this stuff, so the pip version is the same as always have been in > BB time. Let's get OSX updated to a recent version. Bug 1003729 might be useful for historical context.
on mdc1 mac os, the pip version is 1.5.5. - Downloaded version 9.0.1 pip version
Hi Dragrom, do we have an update here? I assume tests have to be run to ensure that all is working fine.
Flags: needinfo?(dcrisan)
Testing changes on t-yosemite-r7-394.test.releng.mdc1.mozilla.com. Need to figure out some things and then I'll add the patch for the review
Flags: needinfo?(dcrisan)
Status: NEW → ASSIGNED
The pip was upgraded: [root@t-yosemite-r7-394.test.releng.mdc1.mozilla.com ~]# /tools/buildbot/bin/pip --version pip 9.0.1 from /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7) pip fail to install packages: /tools/buildbot/bin/pip install --no-deps --no-index --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/packages zope.interface==3.6.1 Collecting zope.interface==3.6.1 The repository located at releng-puppet1.srv.releng.mdc1.mozilla.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host releng-puppet1.srv.releng.mdc1.mozilla.com'. The repository located at releng-puppet2.srv.releng.mdc1.mozilla.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host releng-puppet2.srv.releng.mdc1.mozilla.com'. The repository located at releng-puppet1.srv.releng.usw2.mozilla.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host releng-puppet1.srv.releng.usw2.mozilla.com'. The repository located at releng-puppet1.srv.releng.scl3.mozilla.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host releng-puppet1.srv.releng.scl3.mozilla.com'. The repository located at releng-puppet2.srv.releng.scl3.mozilla.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host releng-puppet2.srv.releng.scl3.mozilla.com'. The repository located at releng-puppet1.srv.releng.use1.mozilla.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host releng-puppet1.srv.releng.use1.mozilla.com'. HTTP error 403 while getting http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/) Could not install requirement zope.interface==3.6.1 from http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz because of error 403 Client Error: Forbidden for url: http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz Could not install requirement zope.interface==3.6.1 from http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz because of HTTP error 403 Client Error: Forbidden for url: http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz for URL http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||simplejson==2.1.3]/Exec[pip /tools/buildbot||simplejson==2.1.3]/returns: No matching distribution found for simplejson==2.1.3
I'm sure the new version of pip is enforcing the use of ssl. So you should be able to just change the repos to use https instead of http.
I had run into these 403's in bug 1373338 too. There is an old bug 1220826, where Hal suggested we switch to https.
Depends on: 1390946
Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Collecting buildbot==0.8.4-pre-moz11 Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Downloading http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/buildbot-0.8.4-pre-moz11.tar.gz (4.1MB) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Exception: Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Traceback (most recent call last): Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: status = self.run(options, args) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 324, in run Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: requirement_set.prepare_files(finder) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 380, in prepare_files Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: ignore_dependencies=self.ignore_dependencies)) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 620, in _prepare_file Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: session=self.session, hashes=hashes) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 821, in unpack_url Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: hashes=hashes Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 659, in unpack_http_url Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: hashes) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 882, in _download_http_url Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: _download_url(resp, link, content_file, hashes) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 605, in _download_url Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: consume(downloaded_chunks) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 852, in consume Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: deque(iterator, maxlen=0) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 571, in written_chunks Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: for chunk in chunks: Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/ui.py", line 139, in iter Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: for x in it: Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 560, in resp_read Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: decode_content=False): Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/response.py", line 357, in stream Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: data = self.read(amt=amt, decode_content=decode_content) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/response.py", line 324, in read Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: flush_decoder = True Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/python27/lib/python2.7/contextlib.py", line 35, in __exit__ Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: self.gen.throw(type, value, traceback) Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/response.py", line 237, in _error_catcher Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: raise ReadTimeoutError(self._pool, None, 'Read timed out.') Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: ReadTimeoutError: HTTPConnectionPool(host='releng-puppet1.srv.releng.mdc1.mozilla.com', port=80): Read timed out. Error: /tools/buildbot/bin/pip install --no-deps --no-index --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.mdc1.mozilla.com --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages --trusted-host releng-puppet2.srv.releng.mdc1.mozilla.com --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.scl3.mozilla.com --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.usw2.mozilla.com --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.use1.mozilla.com buildbot==0.8.4-pre-moz11 returned 2 instead of one of [0] Error: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: change from notrun to 0 failed: /tools/buildbot/bin/pip install --no-deps --no-index --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.mdc1.mozilla.com --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages --trusted-host releng-puppet2.srv.releng.mdc1.mozilla.com --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.scl3.mozilla.com --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.usw2.mozilla.com --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.use1.mozilla.com buildbot==0.8.4-pre-moz11 returned 2 instead of one of [0] When I run the command '/tools/buildbot/bin/pip install --no-deps --no-index --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.mdc1.mozilla.com --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages --trusted-host releng-puppet2.srv.releng.mdc1.mozilla.com --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.scl3.mozilla.com --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.usw2.mozilla.com --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/packages --trusted-host releng-puppet1.srv.releng.use1.mozilla.com buildbot==0.8.4-pre-moz11' from the command line, I received the following error: "Collecting buildbot==0.8.4-pre-moz11 Downloading http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/buildbot-0.8.4-pre-moz11.tar.gz (4.1MB) 7% |██▎ | 286kB 3.2MB/s eta 0:00:02Exception: Traceback (most recent call last): File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 324, in run requirement_set.prepare_files(finder) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 380, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 620, in _prepare_file session=self.session, hashes=hashes) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 821, in unpack_url hashes=hashes File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 659, in unpack_http_url hashes) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 882, in _download_http_url _download_url(resp, link, content_file, hashes) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 605, in _download_url consume(downloaded_chunks) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 852, in consume deque(iterator, maxlen=0) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 571, in written_chunks for chunk in chunks: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/ui.py", line 139, in iter for x in it: File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 560, in resp_read decode_content=False): File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/response.py", line 357, in stream data = self.read(amt=amt, decode_content=decode_content) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/response.py", line 324, in read flush_decoder = True File "/tools/python27/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/response.py", line 237, in _error_catcher raise ReadTimeoutError(self._pool, None, 'Read timed out.') ReadTimeoutError: HTTPConnectionPool(host='releng-puppet1.srv.releng.mdc1.mozilla.com', port=80): Read timed out. " This happened only for releng-puppet hosts from mdc1.
to update the pip version, first we need to remove /tools/buildbot directory. The pip come as tar.gz archive and the version is set in python::virtualenv::settings class. Is expected this behavior?
Flags: needinfo?(jwatkins)
Update the pip version from 1.5.5 to 9.0.1 Added mirror servers to trusted-hosts (added --trusted-host option to pip command), to allow pip to install packages from http
Attachment #8898792 - Flags: review?(jwatkins)
(In reply to Dragos Crisan [:dragrom] from comment #7) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Collecting > buildbot==0.8.4-pre-moz11 > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Downloading > http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/buildbot-0. > 8.4-pre-moz11.tar.gz (4.1MB) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Exception: > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: Traceback (most recent > call last): > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > basecommand.py", line 215, in main > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: status = > self.run(options, args) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > commands/install.py", line 324, in run > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: > requirement_set.prepare_files(finder) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/ > req_set.py", line 380, in prepare_files > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: > ignore_dependencies=self.ignore_dependencies)) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/ > req_set.py", line 620, in _prepare_file > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: > session=self.session, hashes=hashes) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 821, in unpack_url > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: hashes=hashes > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 659, in unpack_http_url > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: hashes) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 882, in _download_http_url > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: _download_url(resp, > link, content_file, hashes) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 605, in _download_url > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: > consume(downloaded_chunks) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/ > __init__.py", line 852, in consume > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: deque(iterator, > maxlen=0) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 571, in written_chunks > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: for chunk in chunks: > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/ > ui.py", line 139, in iter > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: for x in it: > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 560, in resp_read > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: > decode_content=False): > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/ > requests/packages/urllib3/response.py", line 357, in stream > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: data = > self.read(amt=amt, decode_content=decode_content) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/ > requests/packages/urllib3/response.py", line 324, in read > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: flush_decoder = True > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/python27/lib/python2.7/contextlib.py", line 35, in __exit__ > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: > self.gen.throw(type, value, traceback) > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/ > requests/packages/urllib3/response.py", line 237, in _error_catcher > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: raise > ReadTimeoutError(self._pool, None, 'Read timed out.') > Notice: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: ReadTimeoutError: > HTTPConnectionPool(host='releng-puppet1.srv.releng.mdc1.mozilla.com', > port=80): Read timed out. > Error: /tools/buildbot/bin/pip install --no-deps --no-index > --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.mdc1.mozilla.com > --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/ > packages --trusted-host releng-puppet2.srv.releng.mdc1.mozilla.com > --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.scl3.mozilla.com > --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.usw2.mozilla.com > --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/ > packages --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com > --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.use1.mozilla.com > buildbot==0.8.4-pre-moz11 returned 2 instead of one of [0] > Error: > /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python:: > Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip > /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: change from notrun to 0 > failed: /tools/buildbot/bin/pip install --no-deps --no-index > --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.mdc1.mozilla.com > --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/ > packages --trusted-host releng-puppet2.srv.releng.mdc1.mozilla.com > --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.scl3.mozilla.com > --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.usw2.mozilla.com > --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/ > packages --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com > --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.use1.mozilla.com > buildbot==0.8.4-pre-moz11 returned 2 instead of one of [0] > > When I run the command '/tools/buildbot/bin/pip install --no-deps --no-index > --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.mdc1.mozilla.com > --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/ > packages --trusted-host releng-puppet2.srv.releng.mdc1.mozilla.com > --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.scl3.mozilla.com > --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.usw2.mozilla.com > --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/ > packages --trusted-host releng-puppet2.srv.releng.scl3.mozilla.com > --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/ > packages --trusted-host releng-puppet1.srv.releng.use1.mozilla.com > buildbot==0.8.4-pre-moz11' from the command line, I received the following > error: > > "Collecting buildbot==0.8.4-pre-moz11 > Downloading > http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages/buildbot-0. > 8.4-pre-moz11.tar.gz (4.1MB) > 7% |██▎ | 286kB 3.2MB/s eta 0:00:02Exception: > Traceback (most recent call last): > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > basecommand.py", line 215, in main > status = self.run(options, args) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > commands/install.py", line 324, in run > requirement_set.prepare_files(finder) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/ > req_set.py", line 380, in prepare_files > ignore_dependencies=self.ignore_dependencies)) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/ > req_set.py", line 620, in _prepare_file > session=self.session, hashes=hashes) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 821, in unpack_url > hashes=hashes > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 659, in unpack_http_url > hashes) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 882, in _download_http_url > _download_url(resp, link, content_file, hashes) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 605, in _download_url > consume(downloaded_chunks) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/ > __init__.py", line 852, in consume > deque(iterator, maxlen=0) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 571, in written_chunks > for chunk in chunks: > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/ > ui.py", line 139, in iter > for x in it: > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/ > download.py", line 560, in resp_read > decode_content=False): > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/ > requests/packages/urllib3/response.py", line 357, in stream > data = self.read(amt=amt, decode_content=decode_content) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/ > requests/packages/urllib3/response.py", line 324, in read > flush_decoder = True > File "/tools/python27/lib/python2.7/contextlib.py", line 35, in __exit__ > self.gen.throw(type, value, traceback) > File > "/tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/ > requests/packages/urllib3/response.py", line 237, in _error_catcher > raise ReadTimeoutError(self._pool, None, 'Read timed out.') > ReadTimeoutError: > HTTPConnectionPool(host='releng-puppet1.srv.releng.mdc1.mozilla.com', > port=80): Read timed out. > " > This happened only for releng-puppet hosts from mdc1. This is pretty strange and seems like a networking issue. The other possibility that this run was interrupted from the apache reset which happens every time the puppetmaster pulls a new rev from the repos and updates. https://hg.mozilla.org/build/puppet/file/tip/modules/puppetmaster/templates/update.sh.erb#l77
Flags: needinfo?(jwatkins)
(In reply to Dragos Crisan [:dragrom] from comment #8) > to update the pip version, first we need to remove /tools/buildbot > directory. The pip come as tar.gz archive and the version is set in > python::virtualenv::settings class. > Is expected this behavior? I'm not sure why /tools/buildbot needs to be removed in order for pip to be upgraded? Do you have more information here? Does it throw an error? Having to remove /tools/buildbot would be problematic and is seems to me that we have already run into that problem before while trying to upgrade python on osx hosts. And the last solution there was to reimage all the osx hosts. https://bugzilla.mozilla.org/show_bug.cgi?id=1315977#c77
(In reply to Dragos Crisan [:dragrom] from comment #4) :jbarnell: It looks like we're still seeing issues with mdc1 connections. Can you please just put in an allow all as originally requested. This is causing too many issues.
Flags: needinfo?(jbarnell)
Comment on attachment 8898792 [details] [diff] [review] Bug_1388816_Update_pip_on_OSX.patch Review of attachment 8898792 [details] [diff] [review]: ----------------------------------------------------------------- Unless https just doesn't work for some reason, we really should be using https instead of http with --trusted-host. If that is the case, then consider this an r+ and land it. But if https does work, please resubmit for review with https. ::: modules/python/manifests/virtualenv/package.pp @@ +17,5 @@ > $pip_check_py = $python::pip_check_py::file > # give a --find-links option for each data server, so pip will search them all. > $data_server = $config::data_server > $data_servers = $config::data_servers > + # Added mirror_server as trusted hosts, to unloc pip upgrade tests Drop this comment, it isn't needed @@ +22,3 @@ > $pip_options = inline_template("--no-deps --no-index <% > servers = [ @data_server ] + Array(@data_servers) > +servers.uniq.each do |mirror_server| -%> --find-links=http://<%= mirror_server %>/python/packages --trusted-host <%= mirror_server %> <% Is there a reason not to just change this to https instead? Switching it to https would be the better option here.
Attachment #8898792 - Flags: review?(jwatkins) → review-
I tried to use https instead of http. I received the following errors: [root@t-yosemite-r7-394.test.releng.mdc1.mozilla.com ~]# /tools/buildbot/bin/pip install --no-deps --no-index --find-links=https://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages --find-links=https://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages --find-links=https://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages --find-links=https://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages --find-links=https://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages --find-links=https://releng-puppet1.srv.releng.use1.mozilla.com/python/packages zope.interface==3.6.1Collecting zope.interface==3.6.1 /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning. SNIMissingWarning /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Could not fetch URL https://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - skipping /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Could not fetch URL https://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - skipping /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Could not fetch URL https://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - skipping /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Could not fetch URL https://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - skipping /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Could not fetch URL https://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - skipping /tools/buildbot/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Could not fetch URL https://releng-puppet1.srv.releng.use1.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed - skipping HTTP error 403 while getting http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/) Could not install requirement zope.interface==3.6.1 from http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz because of error 403 Client Error: Forbidden for url: http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz Could not install requirement zope.interface==3.6.1 from http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz because of HTTP error 403 Client Error: Forbidden for url: http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz for URL http://pypi.pvt.build.mozilla.org/pub/zope.interface-3.6.1.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/) Looks like a problem with the certificate
Long story short: - updated python from 2.7.3 to 2.7.12 - updated pip to 9.0.1 - changed pypi.pvt and mirror-hosts to use https instead of http Issues: Could not fetch URL https://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping Could not fetch URL https://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping Could not fetch URL https://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping Could not fetch URL https://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping Could not fetch URL https://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping Could not fetch URL https://releng-puppet1.srv.releng.use1.mozilla.com/python/packages: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping This issue appear because pip use a wrong ca file: [root@t-yosemite-r7-394.test.releng.mdc1.mozilla.com ~]# python -c "import ssl; print(ssl.get_default_verify_paths())"DefaultVerifyPaths(cafile=None, capath='/System/Library/OpenSSL/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/System/Library/OpenSSL/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/System/Library/OpenSSL/certs') There are 2 ways to resolve this issue: 1: to add --cert <PATH TO CERT> option to the pip command 2: (That I used) To add the ca path to PIP_CERT variable: export PIP_CERT=/private/var/lib/puppet/ssl/certs/ca.pem The result: [root@t-yosemite-r7-394.test.releng.mdc1.mozilla.com ~]# puppet agent -t --environment=dcrisan --server=releng-puppet2.srv.releng.scl3.mozilla.com Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for t-yosemite-r7-394.test.releng.mdc1.mozilla.com Info: Applying configuration version 'e8225fcb1bbc+' Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/File[/tools/buildbot]/ensure: created Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Exec[virtualenv /tools/buildbot]/returns: executed successfully Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot-slave==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot-slave==0.8.4-pre-moz11]/returns: executed successfully Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||buildbot==0.8.4-pre-moz11]/Exec[pip /tools/buildbot||buildbot==0.8.4-pre-moz11]/returns: executed successfully Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||Twisted==10.2.0]/Exec[pip /tools/buildbot||Twisted==10.2.0]/returns: executed successfully Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||simplejson==2.1.3]/Exec[pip /tools/buildbot||simplejson==2.1.3]/returns: executed successfully Notice: /Stage[main]/Buildslave::Install/Python::Virtualenv[/tools/buildbot]/Python::Virtualenv::Package[/tools/buildbot||zope.interface==3.6.1]/Exec[pip /tools/buildbot||zope.interface==3.6.1]/returns: executed successfully Notice: Finished catalog run in 73.66 seconds
The firewall logs don't seem to support this being an issue with blocking traffic.
Flags: needinfo?(jbarnell)
Use hhtps instead http to access pypi repo Upgrade python from 2.7.3-1 to 2.7.12-1 upgrade pip from 1.5.5 to 9.0.1 Added PIP_CERT env variable. This is required in order to access pip repo using https
Attachment #8898792 - Attachment is obsolete: true
Attachment #8902699 - Flags: review?(jwatkins)
I can confirm, to update pop not need to update python.
Use hhtps instead http to access pypi repo Use python from 2.7.3-1 upgrade pip from 1.5.5 to 9.0.1 Added PIP_CERT env variable. This is required in order to access pip repo using https
Attachment #8902699 - Attachment is obsolete: true
Attachment #8902699 - Flags: review?(jwatkins)
Attachment #8903550 - Flags: review?(jwatkins)
Blocks: 1395960
See Also: 1395960 →
On wich OSX machines need to update pip? TC, BB or both? All versions of OSX or only some versions?
Flags: needinfo?(klibby)
Comment on attachment 8903550 [details] [diff] [review] Bug_1388816_Update_pip_on_OSX.patch Review of attachment 8903550 [details] [diff] [review]: ----------------------------------------------------------------- I'm not sure we should be assuming all OSX should be upgraded to a newer version of pip (although that might be the case). Maybe instead of using the $operatingsystem as a case everywhere, you could use a more exclusive variable that gets set in moco-config.pp such as $pip_version. Then you could set that based on either $operatingsystem or maybe regex of $fqdn. As noted inline, I'm confused about the changes to virtualenv. Are we also upgrading python at the same time? Sorry for all the questions. ::: modules/packages/manifests/virtualenv.pp @@ +34,5 @@ > + 'packages::virtualenv::end': ; > + } > + > + $virtualenv_path = '/tools/virtualenv' > + $py_require = Class['packages::mozilla::python27'] Why set these variable if they are only going to be referenced once? It easier to read if you just write them as a constants in there proper place. eg. python::virtualenv { '/tools/virtualenv': python => $::packages::mozilla::python27::python, require => Class['packages::mozilla::python27']; More importantly, I'm also confused as to why this seems to be creating a virtualenv environment as opposed to installing it. Am I reading that correctly? Does virtualenv get installed with packages::mozilla::python27 ? ::: modules/python/manifests/virtualenv/package.pp @@ +18,5 @@ > # give a --find-links option for each data server, so pip will search them all. > $data_server = $config::data_server > $data_servers = $config::data_servers > + case $::operatingsystem { > + Darwin: { This case has a lot of repeating code where the only difference is http vs https within the inline template string. Instead of including all that code with the case, I would simply use the case to set <%= protocol %> variable to either 'http' or 'https' eg. --find-links=<%= protocol %>://<%= mirror_server %>/python/packages @@ +63,5 @@ > + user => $user, > + environment => [ > + "HOME=${home_dir}", > + 'PIP_CONFIG_FILE=/dev/null', # because sudo will sometimes lead pip to ~administrator/.pip > + 'PIP_CERT=/var/lib/puppet/ssl/certs/ca.pem' # point to ca.certificate to use https instead of http There is a whole lot of code being repeated here also. Is there are reason we can't just include the PIP_CERT env var in all cases and therefore drop the case switch all together? It seems to me the PIP_CERT would have no effect on a non-ssl request. If it is an issue, you could create a case before this exec, that simply sets a variable that is either 'PIP_CERT=/var/lib/puppet/ssl/certs/ca.pem' or ''
Attachment #8903550 - Flags: review?(jwatkins) → review-
(In reply to Dragos Crisan [:dragrom] from comment #20) > On wich OSX machines need to update pip? TC, BB or both? All versions of > OSX or only some versions? Definitely all buildbot OSX hosts. You should also check with Taskcluster to see what version they're using; if it's old, it should also be upgraded (but I *think* it's relatively recent).
Flags: needinfo?(klibby)
No Taskcluster workers seem to use the same technique to prepare the OS. So it's also using a very outdated pip. See bug 1387970 for tests which only run in Taskcluster.
Dustin, how is pip installed on tasckcluster workers? In buildbot I saw that pip is installed in the same time with virtual environment, but in tasckcluster I didn't find the same logic.
Flags: needinfo?(dustin)
Maybe bug 1387970 comment 12 ff. already gives the answer?
Pip is not installed systemwide . Rather, it is installed as part of creation of a virtualenv. So this isn't really a system-installation issue. Also, note that Taskcluster workers are configured with puppet just like Buildbot workers, so fundamentally this isn't something the TC team has any expertise in. It just happens I know a bit about this because I did some work to untangle a mess in bug 1335353. In that bug, I discovered that we were using /tools/buildbot/bin/python (where /tools/buildbot is a virtualenv) and /tools/misc-python/virtualenv.py (which is an ancient version of virtualenv installed for Puppet). Neither of those was ever meant to be used for automation, so it's not surprising that it was old and out of date. Having fixed that bug, we now use /tools/buildbot/bin/virtualenv, which is version 1.7.1.2, which is the version that ships with Ubuntu 12.04 as well. That seems to have pip 1.1 bundled: [root@t-yosemite-r7-0115.test.releng.scl3.mozilla.com ~]# find /tools/virtualenv/ -name pip* /tools/virtualenv//lib/python2.7/site-packages/virtualenv_support/pip-1.1.tar.gz I think that the appropriate way to solve this is to install the desired version of pip into virtualenvs when they are created (/path/to/virtualenv/bin/pip install -U pip==...). Virtualenv creation takes place in mozharness during task execution. Ideally, we would use the in-tree vendored virtualenv for tests, as we do for builds. But tests run without a source checkout, so that virtualenv is unavailable and they must fall back to the version of virtualenv installed systemwide.
Henrik, I think mozharness modifications fall closer to your purview than in Dragos' -- can you fix this?
Flags: needinfo?(dustin) → needinfo?(hskupin)
(In reply to Dustin J. Mitchell [:dustin] from comment #27) > Henrik, I think mozharness modifications fall closer to your purview than in > Dragos' -- can you fix this? I don't work on mozharness code but maybe Aki knows who could work on it.
Flags: needinfo?(hskupin) → needinfo?(aki)
putting a new version of pip on the pypi mirror broke production windows builds (see bug 1399151). I've removed the copy of pip-9.0.1.tar.gz to fix this (I saw that dragos already had a copy in his homedir)
Mozharness uses this function https://dxr.mozilla.org/mozilla-central/rev/f9a5e9ed62103c84e4cde915f4d08f1ce71be83e/testing/mozharness/mozharness/base/python.py#325 to create virtualenvs. I believe this uses the system pip, but setuptools has been known to ping pypi directly (see bug 1369250). This calls https://dxr.mozilla.org/mozilla-central/source/third_party/python/virtualenv/virtualenv.py . Since you're looking at osx, I think pip/python come from puppet.
Flags: needinfo?(aki)
I'm not reading that code the same way you are. For testers (with no topsrcdir), it uses the system *virtualenv*. It then uses `query_python_path` to look for binaries *inside* that virtualenv -- pip, in this case. Virtualenv installs pip in one of three ways: 1. finds a pip tarball in the current directory 2. was installed with a pip tarball 3. pulls it from pypi.python.org Note that none of those is "use a pip binary found in PATH" - a virtualenv will never use the system pip. In this case, #2 is the selected option, and per comment 26 the version of pip it installs is pip-1.1. This bit here: https://dxr.mozilla.org/mozilla-central/rev/f9a5e9ed62103c84e4cde915f4d08f1ce71be83e/testing/mozharness/mozharness/base/python.py#415 for module in ('distribute', 'pip'): if c.get('%s_url' % module): self.download_file(c['%s_url' % module], parent_dir=dirs['abs_work_dir']) looks for distribute and pip tarballs, and downloads them where virtualenv will find them, to support #1 above. So I think the right solution is to put a newer version of pip on pypi.pub.b.m.o and set pip_url to point to it in all of the mozharness config files. It's probably worth setting distribute_url too. This puts control of pip versions nicely in-tree, so relops will never have to worry about this again. And, once we get sparse checkouts on testers, we will use the in-tree virtualenv too, meaning even the virtualenv version will be controlled in-tree. Yay! Per comment 29, just *uploading* a newer pip to pypi.pub.b.m.o causes windows tasks to fail. That's obviously a bug with the windows tasks, but suggests there are other demons to be found in uploading things to pypi. So, that's something for relops to worry about :(
Uploading it causes it to fail because the in-tree code that installs pip into the virtualenv looks like it specifies > 1.5 for everything. I think the problem is the in-tree code that assumes you want the same version of pip (or anything, really) across all OSes. In order for this to not be problematic, the in-tree code would need to be modified so that it was OS dependent.
Which in-tree code is that?
aki do you have a dxr link to the stuff you were looking at last night?
Flags: needinfo?(aki)
(In reply to Amy Rich [:arr] [:arich] from comment #34) > aki do you have a dxr link to the stuff you were looking at last night? I had that tab open, so: https://dxr.mozilla.org/mozilla-central/search?q=pip%3E%3D1.5
Flags: needinfo?(aki)
This code will have to work for both tc and buildbot, plus any people invoking it outside of automation (not sure if mach calls the script or bypasses mozharness entirely). I'm not sure why we want to upgrade pip now. I think it's safest to leave moz's pypi alone for packages like pip. Some thoughts: - updating puppet but not pypi might work to target specific platforms. Might be worth testing to see if a machine with a system pip 9.0.1 gets pip 9.0.1 or 1.5.5 in its mozharness virtualenvs - i still think getting a valid ssl cert on moz's pypi and changing the pypirc's to point to https rather than http will fix this pip 9.0.1 issue. might be worth testing on a windows tester: install pip 9.0.1 in a virtualenv, update the pypirc to https (pointing at a server with a valid cert) and seeing if pip installs work. Py2's ssl validation tends to be *weaker* than py3, so in theory it should accept a wider range of certs (assuming non-modern algs) - trying to pin automation limitations in the code is hacky and will affect more than we want, unless we check for is_automation and if TASKID is in env... then we can target buildbot, ish. doing it via config is better but means we need to know which configs are used by all buildbot and taskcluster jobs. I think the valid server ssl cert seems like the easiest solution, but I might be wrong.
(In reply to Aki Sasaki [:aki] from comment #36) > - trying to pin automation limitations in the code is hacky and will affect > more than we want, unless we check for is_automation and if TASKID is in > env... then we can target buildbot, ish. doing it via config is better but > means we need to know which configs are used by all buildbot and taskcluster > jobs. We also have to uplift to beta, release, and esr for the code solution... not a biggy, just a reminder.
See Also: → 1399926
So based on what we discovered here, I think this bug is WONTFIX. The system pip is not the one used in the venv, so updating it shouldn't have any effect on failing tests using that venv. That takes us back to bug 1297515 where someone needs to modify the in-tree configuration to install the right version of pip into the venv and/or pin it per platform. I also think we should investigate getting a valid cert on the pypi mirror as aki suggests above. I'll open a separate bug for that. That might help facilitate upgrading pip in the venv. I've opened 1399926 to track that.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Blocks: t-yosemite-r7-394
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: