Closed
Bug 1447584
Opened 7 years ago
Closed 7 years ago
Intermittent ConnectionError: HTTPSConnectionPool(host='cloud-mirror-production-us-east-1.s3.amazonaws.com', port=443): Max retries exceeded - Failed to establish a new connection
Categories
(Firefox Build System :: Task Configuration, task, P5)
Firefox Build System
Task Configuration
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 1435107
People
(Reporter: intermittent-bug-filer, Unassigned)
Details
(Keywords: intermittent-failure)
Filed by: ncsoregi [at] mozilla.com
https://treeherder.mozilla.org/logviewer.html#?job_id=169283718&repo=mozilla-central
https://queue.taskcluster.net/v1/task/EcS9rYqsTkK2T5afzTpb_A/runs/0/artifacts/public/logs/live_backing.log
[task 2018-03-20T20:21:15.164Z] 20:21:15 INFO - Error running mach:
[task 2018-03-20T20:21:15.164Z] 20:21:15 INFO - ['artifact', 'toolchain', '-v', '--retry', '4', '--artifact-manifest', '/builds/worker/workspace/build/src/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache', 'public/build/clang.tar.xz@T0sgTTO4RF2TUtP1hHMneg', 'public/build/rustc.tar.xz@aHJQvxAuQ4ynvdPwaqWIMw', 'public/build/sccache2.tar.xz@EOWg-A0hTCq_sr7aPSMx7g']
[task 2018-03-20T20:21:15.164Z] 20:21:15 INFO - The error occurred in code that was called by the mach command. This is either
[task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - a bug in the called code itself or in the way that mach is calling it.
[task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - You should consider filing a bug for this issue.
[task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - If filing a bug, please include the full output of mach, including this error
[task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - message.
[task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - The details of the failure are as follows:
[task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - ConnectionError: HTTPSConnectionPool(host='cloud-mirror-production-us-east-1.s3.amazonaws.com', port=443): Max retries exceeded with url: /https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FT0sgTTO4RF2TUtP1hHMneg%2F0%2Fpublic%2FchainOfTrust.json.asc (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f41ee6041d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
[task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/mach_commands.py", line 1426, in artifact_toolchain
[task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - record = ArtifactRecord(task_id, name)
[task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/mach_commands.py", line 1321, in __init__
[task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - get_artifact_url(task_id, 'public/chainOfTrust.json.asc'))
[task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 480, in get
[task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - return self.request('GET', url, **kwargs)
[task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 468, in request
[task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - resp = self.send(prep, **send_kwargs)
[task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 597, in send
[task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - history = [resp for resp in gen] if allow_redirects else []
[task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 195, in resolve_redirects
[task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - **adapter_kwargs
[task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 576, in send
[task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - r = adapter.send(request, **kwargs)
[task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/adapters.py", line 437, in send
[task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - raise ConnectionError(e, request=request)
[task 2018-03-20T20:21:15.178Z] 20:21:15 ERROR - Return code: 1
[task 2018-03-20T20:21:15.179Z] 20:21:15 ERROR - 1 not in success codes: [0]
[task 2018-03-20T20:21:15.179Z] 20:21:15 WARNING - setting return code to 2
[task 2018-03-20T20:21:15.179Z] 20:21:15 FATAL - Halting on failure while running ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/build/src/mach', 'artifact', 'toolchain', '-v', '--retry', '4', '--artifact-manifest', '/builds/worker/workspace/build/src/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache', 'public/build/clang.tar.xz@T0sgTTO4RF2TUtP1hHMneg', 'public/build/rustc.tar.xz@aHJQvxAuQ4ynvdPwaqWIMw', 'public/build/sccache2.tar.xz@EOWg-A0hTCq_sr7aPSMx7g']
[task 2018-03-20T20:21:15.179Z] 20:21:15 FATAL - Running post_fatal callback...
[task 2018-03-20T20:21:15.179Z] 20:21:15 FATAL - Exiting 2
[task 2018-03-20T20:21:15.179Z] 20:21:15 INFO - [mozharness: 2018-03-20 20:21:15.179689Z] Finished build step (failed)
[task 2018-03-20T20:21:15.179Z] 20:21:15 INFO - Running post-run listener: _summarize
[task 2018-03-20T20:21:15.179Z] 20:21:15 ERROR - # TBPL FAILURE #
[task 2018-03-20T20:21:15.180Z] 20:21:15 INFO - [mozharness: 2018-03-20 20:21:15.180093Z] FxDesktopBuild summary:
[task 2018-03-20T20:21:15.180Z] 20:21:15 ERROR - # TBPL FAILURE #
Comment 1•7 years ago
|
||
cloud-mirror-production-us-east-1 is an AWS system used by TC for serving artifacts, and the job is running on AWS too. Do we have any option other than increasing the number of retries ? The call to mach artifact is ~14 minutes before the error messages so there must already be considerable backoff configured.
Product: Release Engineering → Taskcluster
Comment 2•7 years ago
|
||
It's not a system, just an S3 bucket, and in the same region as the EC2 instance in question. So, basically, yes, increasing the number of retries is the only option here.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
Updated•7 years ago
|
Status: RESOLVED → REOPENED
Component: General → Task Configuration
Product: Taskcluster → Firefox Build System
Resolution: WORKSFORME → ---
Comment hidden (Intermittent Failures Robot) |
Updated•7 years ago
|
Status: REOPENED → RESOLVED
Closed: 7 years ago → 7 years ago
Resolution: --- → DUPLICATE
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
You need to log in
before you can comment on or make changes to this bug.
Description
•