Closed Bug 1452179 Opened 6 years ago Closed 6 years ago

Download failures: Could not install python package: /Users/cltbld/tasks/task_1523030156/build/venv/bin/pip install --timeout 120 --no-index jsonschema==2.5.1 failed after 5 tries!

Categories

(Release Engineering :: General, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Whiteboard: [stockwell infra])

Attachments

(9 files)

45 bytes, text/x-phabricator-request
nthomas
: review+
Details | Review
45 bytes, text/x-phabricator-request
bhearsum
: review+
Details | Review
45 bytes, text/x-phabricator-request
bhearsum
: review+
Details | Review
45 bytes, text/x-phabricator-request
bhearsum
: review+
Details | Review
45 bytes, text/x-phabricator-request
Details | Review
45 bytes, text/x-phabricator-request
Details | Review
45 bytes, text/x-phabricator-request
bhearsum
: review+
Details | Review
46 bytes, text/x-phabricator-request
bhearsum
: review+
Details | Review
46 bytes, text/x-phabricator-request
bhearsum
: review+
Details | Review
10:20:32     INFO -  Ignoring indexes: https://pypi.python.org/simple
10:20:32     INFO -  Collecting jsonschema==2.5.1
10:20:34     INFO -    Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2550>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:34     INFO -    Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2710>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:35     INFO -    Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2890>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:37     INFO -    Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2a10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:41     INFO -    Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2b90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:42     INFO -    Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2610>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:43     INFO -    Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d29d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:44     INFO -    Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2a90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:46     INFO -    Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2750>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:50     INFO -    Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x10b0d2e50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',)': /pub
10:20:50     INFO -    Could not find a version that satisfies the requirement jsonschema==2.5.1 (from versions: )
10:20:50     INFO -  No matching distribution found for jsonschema==2.5.1
10:20:50  WARNING - Return code: 1
10:20:50    FATAL - Could not install python package: /Users/cltbld/tasks/task_1523030156/build/venv/bin/pip install --timeout 120 --no-index jsonschema==2.5.1 failed after 5 tries!
10:20:50    FATAL - Running post_fatal callback...
10:20:50    FATAL - Exiting -1
10:20:50     INFO - Running post-action listener: _resource_record_post_action
10:20:50     INFO - Running post-action listener: _start_resource_monitoring
10:20:50  WARNING - Unable to start resource monitor: Traceback (most recent call last):
10:20:50  WARNING -   File "/Users/cltbld/tasks/task_1523030156/mozharness/mozharness/base/python.py", line 515, in _start_resource_monitoring
10:20:50  WARNING -     from mozsystemmonitor.resourcemonitor import SystemResourceMonitor
10:20:50  WARNING - ImportError: No module named mozsystemmonitor.resourcemonitor
10:20:50     INFO - [mozharness: 2018-04-06 17:20:50.100132Z] Finished create-virtualenv step (failed)
10:20:50     INFO - Running post-run listener: _resource_record_post_run
10:20:50     INFO - Running post-run listener: _upload_blobber_files
10:20:50  WARNING - Blob upload gear skipped. Missing cmdline options.
10:20:50     INFO - Running post-run listener: copy_logs_to_upload_dir
10:20:50     INFO - Copying logs to upload dir...
10:20:50     INFO - mkdir: /Users/cltbld/tasks/task_1523030156/build/upload/logs
[taskcluster 2018-04-06T17:20:50.129Z] Exit Code: 255
It's frequent and across platforms.

Trees are closed for this.
Severity: normal → blocker
Product: Firefox → Taskcluster
Component: General → General Automation
Product: Taskcluster → Release Engineering
QA Contact: catlee
Whiteboard: [stockwell disable-recommended] → [stockwell infra]
Comment on attachment 8965787 [details]
Bug 1452179: Default to only using internal pypi; r=nthomas

Nick Thomas [:nthomas] (UTC+13) has approved the revision.

https://phabricator.services.mozilla.com/D873
Attachment #8965787 - Flags: review+
Attachment #8965787 - Attachment description: Bug 1452179: Default to only using internal pypi. → Bug 1452179: Default to only using internal pypi; r=nthomas
Keywords: leave-open
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/mozilla-central/rev/88e916621af9
Default to only using internal pypi; r=nthomas
We now vendor pip via the vendored copy of virtualenv. Stop trying to install a
specific version of pip elsewhere, particularly stop downgrading it.
Comment on attachment 8968535 [details]
Bug 1452179: Stop pinning pip; r=bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

https://phabricator.services.mozilla.com/D962
Attachment #8968535 - Flags: review+
Backed out for talos failures

backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/33d4420119d61139f2cb7ced1cd73ba72efe8cc2

push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=747d0b814dc1be3e5ac04e080361ab0b0fc034f9&filter-searchStr=talos&group_state=expanded

failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=174148761&repo=mozilla-inbound&lineNumber=549

17:31:35     INFO -  C:\Users\task_1523985266\build/venv/scripts/python: can't open file 'C:\Users\task_1523985266\build/venv/scripts/easy_install-2.7-script.py': [Errno 2] No such file or directory
17:31:35  WARNING - Return code: 2
17:31:35    FATAL - Could not install python package: C:\Users\task_1523985266\build/venv/scripts/python C:\Users\task_1523985266\build/venv/scripts/easy_install-2.7-script.py --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub pywin32 failed after 5 tries!
17:31:35    FATAL - Running post_fatal callback...
17:31:35    FATAL - Exiting -1
17:31:35     INFO - Running post-action listener: _resource_record_post_action
17:31:35     INFO - Running post-action listener: _start_resource_monitoring
17:31:35  WARNING - Unable to start resource monitor: Traceback (most recent call last):
17:31:35  WARNING -   File "C:\Users\task_1523985266\mozharness\mozharness\base\python.py", line 516, in _start_resource_monitoring
17:31:35  WARNING -     from mozsystemmonitor.resourcemonitor import SystemResourceMonitor
17:31:35  WARNING - ImportError: No module named mozsystemmonitor.resourcemonitor
17:31:35     INFO - [mozharness: 2018-04-17 17:31:35.883000Z] Finished create-virtualenv step (failed)
17:31:35     INFO - Running post-run listener: _resource_record_post_run
17:31:35     INFO - Running post-run listener: _upload_blobber_files
17:31:35  WARNING - Blob upload gear skipped. Missing cmdline options.
17:31:35     INFO - Running post-run listener: copy_logs_to_upload_dir
17:31:35     INFO - Copying logs to upload dir...
17:31:35     INFO - mkdir: C:\Users\task_1523985266\build\upload\logs
[taskcluster 2018-04-17T17:31:35.942Z]   Exit Code: 4294967295
[taskcluster 2018-04-17T17:31:35.942Z]   User Time: 15.625ms
[taskcluster 2018-04-17T17:31:35.942Z] Kernel Time: 0s
[taskcluster 2018-04-17T17:31:35.943Z]   Wall Time: 16m54.4399875s
[taskcluster 2018-04-17T17:31:35.943Z] Peak Memory: 5005312
[taskcluster 2018-04-17T17:31:35.943Z]      Result: FAILED
[taskcluster 2018-04-17T17:31:35.943Z] === Task Finished ===
[taskcluster 2018-04-17T17:31:35.943Z] Task Duration: 16m54.4526571s
Flags: needinfo?(mozilla)
Attachment #8968535 - Attachment description: Bug 1452179: Stop pinning pip; r?bhearsum → Bug 1452179: Stop pinning pip; r=bhearsum
New versions of pip don't install entrypoint scripts with a `-script.py` on
windows. Instead, just call the bare script name everywhere. On unix-like
systems, this uses the shebang; on Windows, this executes the `.exe` wrapper.
Attachment #8968689 - Attachment description: Bug 1452179: [mozharness] Don't special case the path to mozinstall on windows; r?nthomas → Bug 1452179: [mozharness] Don't special case the path to mozinstall on windows.
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6e6214abf0f2
Stop pinning pip; r=bhearsum
https://hg.mozilla.org/integration/mozilla-inbound/rev/bf49dbd6d055
[mozharness] Use `python -m easy_install` to run easy_install; r=bhearsum
https://hg.mozilla.org/integration/mozilla-inbound/rev/8bfb70643041
[mozharness] Don't special case the path to mozinstall on windows; r=whimboo
https://hg.mozilla.org/integration/mozilla-inbound/rev/5ddb466ecccf
[mozharness] Don't set virtualenv_modules in talos, since it is unused; r=whimboo
Flags: needinfo?(mozilla)
Comment on attachment 8968608 [details]
Bug 1452179: [mozharness] Use `python -m easy_install` to run easy_install; r=bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

https://phabricator.services.mozilla.com/D964
Attachment #8968608 - Flags: review+
Attachment #8967055 - Attachment description: Bug 1452179: Get rid of pip configs when they match the default; r?nthomas → Bug 1452179: [mozharness] Get rid of pip configs when they match the default; r?bhearsum
Tom, I think I'm missing some context here - can you give me a quick rundown of the patches?
Flags: needinfo?(mozilla)
We have internal pypi mirrors that we want to use for everything. In https://phabricator.services.mozilla.com/D873 (already landed), I switched the default location to install from to be our internal mirror. The remaining patches get rid of the configs that point to that explicitly, as well as those that point to non-public copy of that. Then, since Bug 1399926 is now resolved, I switched to using https and dropped the code supported installing from non-https sources.
Flags: needinfo?(mozilla)
Comment on attachment 8969476 [details]
Bug 1452179: [mozharness] Get rid of references to private pypi server; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

https://phabricator.services.mozilla.com/D999
Attachment #8969476 - Flags: review+
Comment on attachment 8969477 [details]
Bug 1452179: [mozharness] Use https for internal pypi mirror; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

https://phabricator.services.mozilla.com/D1000
Attachment #8969477 - Flags: review+
Comment on attachment 8969478 [details]
Bug 1452179: [mozharness] Remove support for installing from inscure python sources; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

https://phabricator.services.mozilla.com/D1001
Attachment #8969478 - Flags: review+
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8758f6f2e9fb
[mozharness] Get rid of pip configs when they match the default; r=bhearsum
https://hg.mozilla.org/integration/mozilla-inbound/rev/48284706be22
[mozharness] Get rid of references to private pypi server; r=bhearsum
https://hg.mozilla.org/integration/mozilla-inbound/rev/4e9424aae45f
[mozharness] Use https for internal pypi mirror; r=bhearsum
https://hg.mozilla.org/integration/mozilla-inbound/rev/4f8ee709a28b
[mozharness] Remove support for installing from inscure python sources; r=bhearsum
Comment on attachment 8967055 [details]
Bug 1452179: [mozharness] Get rid of pip configs when they match the default; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

https://phabricator.services.mozilla.com/D911
Attachment #8967055 - Flags: review+
Depends on: 1456182
:tomprince, this is marked as leave open, is there more work to do here?
Flags: needinfo?(mozilla)
Component: General Automation → General
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
Flags: needinfo?(mozilla)
You need to log in before you can comment on or make changes to this bug.