Closed Bug 1447584 Opened 7 years ago Closed 7 years ago

Intermittent ConnectionError: HTTPSConnectionPool(host='cloud-mirror-production-us-east-1.s3.amazonaws.com', port=443): Max retries exceeded - Failed to establish a new connection

Categories

(Firefox Build System :: Task Configuration, task, P5)

task

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1435107

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure)

Filed by: ncsoregi [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=169283718&repo=mozilla-central https://queue.taskcluster.net/v1/task/EcS9rYqsTkK2T5afzTpb_A/runs/0/artifacts/public/logs/live_backing.log [task 2018-03-20T20:21:15.164Z] 20:21:15 INFO - Error running mach: [task 2018-03-20T20:21:15.164Z] 20:21:15 INFO - ['artifact', 'toolchain', '-v', '--retry', '4', '--artifact-manifest', '/builds/worker/workspace/build/src/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache', 'public/build/clang.tar.xz@T0sgTTO4RF2TUtP1hHMneg', 'public/build/rustc.tar.xz@aHJQvxAuQ4ynvdPwaqWIMw', 'public/build/sccache2.tar.xz@EOWg-A0hTCq_sr7aPSMx7g'] [task 2018-03-20T20:21:15.164Z] 20:21:15 INFO - The error occurred in code that was called by the mach command. This is either [task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - a bug in the called code itself or in the way that mach is calling it. [task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - You should consider filing a bug for this issue. [task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - If filing a bug, please include the full output of mach, including this error [task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - message. [task 2018-03-20T20:21:15.165Z] 20:21:15 INFO - The details of the failure are as follows: [task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - ConnectionError: HTTPSConnectionPool(host='cloud-mirror-production-us-east-1.s3.amazonaws.com', port=443): Max retries exceeded with url: /https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FT0sgTTO4RF2TUtP1hHMneg%2F0%2Fpublic%2FchainOfTrust.json.asc (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f41ee6041d0>: Failed to establish a new connection: [Errno 110] Connection timed out',)) [task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/mach_commands.py", line 1426, in artifact_toolchain [task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - record = ArtifactRecord(task_id, name) [task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/mach_commands.py", line 1321, in __init__ [task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - get_artifact_url(task_id, 'public/chainOfTrust.json.asc')) [task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 480, in get [task 2018-03-20T20:21:15.166Z] 20:21:15 INFO - return self.request('GET', url, **kwargs) [task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 468, in request [task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - resp = self.send(prep, **send_kwargs) [task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 597, in send [task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - history = [resp for resp in gen] if allow_redirects else [] [task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 195, in resolve_redirects [task 2018-03-20T20:21:15.167Z] 20:21:15 INFO - **adapter_kwargs [task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 576, in send [task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - r = adapter.send(request, **kwargs) [task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - File "/builds/worker/workspace/build/src/third_party/python/requests/requests/adapters.py", line 437, in send [task 2018-03-20T20:21:15.168Z] 20:21:15 INFO - raise ConnectionError(e, request=request) [task 2018-03-20T20:21:15.178Z] 20:21:15 ERROR - Return code: 1 [task 2018-03-20T20:21:15.179Z] 20:21:15 ERROR - 1 not in success codes: [0] [task 2018-03-20T20:21:15.179Z] 20:21:15 WARNING - setting return code to 2 [task 2018-03-20T20:21:15.179Z] 20:21:15 FATAL - Halting on failure while running ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/build/src/mach', 'artifact', 'toolchain', '-v', '--retry', '4', '--artifact-manifest', '/builds/worker/workspace/build/src/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache', 'public/build/clang.tar.xz@T0sgTTO4RF2TUtP1hHMneg', 'public/build/rustc.tar.xz@aHJQvxAuQ4ynvdPwaqWIMw', 'public/build/sccache2.tar.xz@EOWg-A0hTCq_sr7aPSMx7g'] [task 2018-03-20T20:21:15.179Z] 20:21:15 FATAL - Running post_fatal callback... [task 2018-03-20T20:21:15.179Z] 20:21:15 FATAL - Exiting 2 [task 2018-03-20T20:21:15.179Z] 20:21:15 INFO - [mozharness: 2018-03-20 20:21:15.179689Z] Finished build step (failed) [task 2018-03-20T20:21:15.179Z] 20:21:15 INFO - Running post-run listener: _summarize [task 2018-03-20T20:21:15.179Z] 20:21:15 ERROR - # TBPL FAILURE # [task 2018-03-20T20:21:15.180Z] 20:21:15 INFO - [mozharness: 2018-03-20 20:21:15.180093Z] FxDesktopBuild summary: [task 2018-03-20T20:21:15.180Z] 20:21:15 ERROR - # TBPL FAILURE #
cloud-mirror-production-us-east-1 is an AWS system used by TC for serving artifacts, and the job is running on AWS too. Do we have any option other than increasing the number of retries ? The call to mach artifact is ~14 minutes before the error messages so there must already be considerable backoff configured.
Product: Release Engineering → Taskcluster
It's not a system, just an S3 bucket, and in the same region as the EC2 instance in question. So, basically, yes, increasing the number of retries is the only option here.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Component: General → Task Configuration
Product: Taskcluster → Firefox Build System
Resolution: WORKSFORME → ---
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.