Closed Bug 1399151 Opened 7 years ago Closed 7 years ago

Windows buildbot test fail: Could not install python package: C:\slave\test\build\venv\Scripts\pip install --timeout 120 --no-index --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub psutil==3.1.1 failed

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aryx, Assigned: arich)

Details

(Whiteboard: [stockwell infra])

Trees are closed for this permafailure. First push with failures (because pgo build take longer)? https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=3867d0df5cafe68058139c4de8e47c4d6e79e567&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=130376432&repo=autoland 07:27:45 INFO - Ignoring indexes: http://pypi.python.org/simple/ 07:27:45 INFO - Downloading/unpacking psutil>=3.1.1 07:27:45 INFO - Running setup.py egg_info for package psutil 07:27:45 INFO - warning: no previously-included files matching '*' found under directory 'docs\_build' 07:27:45 INFO - warning: manifest_maker: MANIFEST.in, line 18: 'recursive-include' expects <dir> <pattern1> <pattern2> ... 07:27:45 INFO - Installing collected packages: psutil 07:27:45 INFO - Running setup.py install for psutil 07:27:45 INFO - building 'psutil._psutil_windows' extension 07:27:45 INFO - error: Unable to find vcvarsall.bat 07:27:45 INFO - Complete output from command C:\slave\test\build\venv\Scripts\python.exe -c "import setuptools;__file__='C:\\slave\\test\\build\\venv\\build\\psutil\\setup.py';execfile(__file__)" install --single-version-externally-managed --record c:\users\cltbld~1.001\appdata\local\temp\pip-j9izfk-record\install-record.txt --install-headers C:\slave\test\build\venv\include\site\python2.7: 07:27:45 INFO - running install 07:27:45 INFO - running build 07:27:45 INFO - running build_py 07:27:45 INFO - creating build 07:27:45 INFO - creating build\lib.win32-2.7 07:27:45 INFO - creating build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_common.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_compat.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_psbsd.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_pslinux.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_psosx.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_psposix.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_pssunos.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\_pswindows.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - copying psutil\__init__.py -> build\lib.win32-2.7\psutil 07:27:45 INFO - running build_ext 07:27:45 INFO - building 'psutil._psutil_windows' extension 07:27:45 INFO - error: Unable to find vcvarsall.bat 07:27:45 INFO - ---------------------------------------- 07:27:45 INFO - Command C:\slave\test\build\venv\Scripts\python.exe -c "import setuptools;__file__='C:\\slave\\test\\build\\venv\\build\\psutil\\setup.py';execfile(__file__)" install --single-version-externally-managed --record c:\users\cltbld~1.001\appdata\local\temp\pip-j9izfk-record\install-record.txt --install-headers C:\slave\test\build\venv\include\site\python2.7 failed with error code 1 07:27:45 INFO - Storing complete log in C:\Users\cltbld.T-W864-IX-094.001\AppData\Roaming\pip\pip.log 07:27:45 WARNING - Return code: 1 07:27:45 WARNING - Could not install python package: C:\slave\test\build\venv\Scripts\pip install --timeout 120 --no-index --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub psutil>=3.1.1 failed after 1 tries! See bug 1351353 for the previously intermittent "error: Unable to find vcvarsall.bat".
It looks like someone uploaded a new version of pip that broke things: http://pypi.pvt.build.mozilla.org/pub/pip-9.0.1.tar.gz 12-Sep-2017 07:14 1.1M I've removed that and verified that it's synced out to all of the mirrors. We can reopen the trees now.
Note: if machines still have the old pip installed in the venv, they need to clean up the venv or will still fail.
Whiteboard: [stockwell infra]
We're still frequently hitting these failures four hours later. Is there anything we could do to kill these instances and purge their venv?
Flags: needinfo?(arich)
From discussion on irc, aki thinks that any new job should be removing and reinstalling the venv, but that pip has a local cache that keeps installing the newer, broken version, even though it's been removed from the pypi server.
Flags: needinfo?(arich)
It turns out that clearing it from the puppet server does not clear it from the pypi mirror. I've logged into the relengweb cluster and removed it from /mnt/netapp/relengweb/pypi/pub and verified that it's gone from http://pypi.pub.build.mozilla.org/pub/
It looks like deleting it from the pypi mirror fixed things. We're seeing green tests now. KWierso: please RF when you're happy that we're out of the woods.
Assignee: nobody → arich
Flags: needinfo?(wkocher)
Things seem to be getting greener, but I ran out of day to keep watching things to make that call for sure.
Flags: needinfo?(wkocher)
philor pointed out some more failures and it turns out the proxxy servers in usw2 still had pip 9.0.1, eg http://pypi.pvt.build.mozilla.org.proxxy1.srv.releng.usw2.mozilla.com/pub/. use1 wasn't affected by this. I did find /var/cache/proxxy/pypi.{pub,pvt}.build.mozilla.org -type f -delete on proxxy1.srv.releng.usw2.mozilla.com to resolve.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.