Closed Bug 1259263 Opened 10 years ago Closed 10 years ago

Windows builds failing during pip install

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
Unspecified
Windows
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: KWierso, Unassigned)

References

Details

I'm seeing failures like https://treeherder.mozilla.org/logviewer.html#?job_id=24470053&repo=mozilla-inbound across multiple trees. Closing most of the trees I care about.
15:22:50 INFO - Creating virtualenv c:\builds\moz2_slave\m-in-w32-000000000000000000000\build\venv 15:22:50 INFO - mkdir: c:\builds\moz2_slave\m-in-w32-000000000000000000000\build 15:22:50 INFO - Running command: ['c:\\mozilla-build\\python27\\python.exe', 'c:/mozilla-build/buildbotve/virtualenv.py', '--no-site-packages', '--distribute', 'c:\\builds\\moz2_slave\\m-in-w32-000000000000000000000\\build\\venv'] in c:\builds\moz2_slave\m-in-w32-000000000000000000000\build 15:22:50 INFO - Copy/paste: c:\mozilla-build\python27\python.exe c:/mozilla-build/buildbotve/virtualenv.py --no-site-packages --distribute c:\builds\moz2_slave\m-in-w32-000000000000000000000\build\venv 15:22:54 INFO - The --no-site-packages flag is deprecated; it is now the default behavior. 15:22:54 INFO - New python executable in c:\builds\moz2_slave\m-in-w32-000000000000000000000\build\venv\Scripts\python.exe 15:23:00 INFO - Installing distribute................................................................................................................................................................................................done. 15:23:26 INFO - Installing pip...... 15:23:26 INFO - Complete output from command c:\builds\moz2_slave...v\Scripts\python.exe c:\builds\moz2_slave...sy_install-script.py pip: 15:23:26 INFO - Searching for pip 15:23:26 INFO - Reading http://pypi.python.org/simple/pip/ 15:23:26 INFO - error: An existing connection was forcibly closed by the remote host 15:23:26 INFO - ---------------------------------------- 15:23:26 INFO - ...Installing pip...done. 15:23:26 ERROR - Traceback (most recent call last): 15:23:26 INFO - File "c:/mozilla-build/buildbotve/virtualenv.py", line 2270, in <module> 15:23:26 INFO - main() 15:23:26 INFO - File "c:/mozilla-build/buildbotve/virtualenv.py", line 928, in main 15:23:26 INFO - never_download=options.never_download) 15:23:26 INFO - File "c:/mozilla-build/buildbotve/virtualenv.py", line 1042, in create_environment 15:23:26 INFO - install_pip(py_executable, search_dirs=search_dirs, never_download=never_download) 15:23:26 INFO - File "c:/mozilla-build/buildbotve/virtualenv.py", line 640, in install_pip 15:23:26 INFO - filter_stdout=_filter_setup) 15:23:26 INFO - File "c:/mozilla-build/buildbotve/virtualenv.py", line 1006, in call_subprocess 15:23:26 INFO - % (cmd_desc, proc.returncode)) 15:23:26 INFO - OSError: Command c:\builds\moz2_slave...v\Scripts\python.exe c:\builds\moz2_slave...sy_install-script.py pip failed with error code 1 15:23:26 ERROR - Return code: 1 15:23:26 ERROR - 1 not in success codes: [0] 15:23:26 ERROR - failures found while parsing output 15:23:26 WARNING - setting return code to 2 15:23:26 FATAL - Halting on failure while running ['c:\\mozilla-build\\python27\\python.exe', 'c:/mozilla-build/buildbotve/virtualenv.py', '--no-site-packages', '--distribute', 'c:\\builds\\moz2_slave\\m-in-w32-000000000000000000000\\build\\venv'] 15:23:26 FATAL - Running post_fatal callback... 15:23:26 FATAL - Exiting 2
From IRC - we're using http://pypi.python.org to install pip, and they're status page says 'Partially Degraded Service' on the back of the Fastly CDN having 'Degraded Performance' in LAX and SJC. markco confirmed that traffic is routed back over the VPC to SCL3 then out to the internet, so that's our smoking gun. More generally, gps makes a good point - we should be using the internal pypi, or if the virtualenv we have on windows doesn't allow that then --never-download or something.
The Python Status page says that everything is operating normally now.
Closing this since the crisis appears to have been averted. Further work can happen in bug 1148087.
Status: NEW → RESOLVED
Closed: 10 years ago
Depends on: 1148087
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.