1447584 - Intermittent ConnectionError: HTTPSConnectionPool(host='cloud-mirror-production-us-east-1.s3.amazonaws.com', port=443): Max retries exceeded - Failed to establish a new connection

Status: RESOLVED DUPLICATE of bug 1435107

(Firefox Build System :: Task Configuration, task, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=169283718&repo=mozilla-central

https://queue.taskcluster.net/v1/task/EcS9rYqsTkK2T5afzTpb_A/runs/0/artifacts/public/logs/live_backing.log

[task 2018-03-20T20:21:15.164Z] 20:21:15     INFO -  Error running mach:
[task 2018-03-20T20:21:15.164Z] 20:21:15     INFO -      ['artifact', 'toolchain', '-v', '--retry', '4', '--artifact-manifest', '/builds/worker/workspace/build/src/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache', 'public/build/clang.tar.xz@T0sgTTO4RF2TUtP1hHMneg', 'public/build/rustc.tar.xz@aHJQvxAuQ4ynvdPwaqWIMw', 'public/build/sccache2.tar.xz@EOWg-A0hTCq_sr7aPSMx7g']
[task 2018-03-20T20:21:15.164Z] 20:21:15     INFO -  The error occurred in code that was called by the mach command. This is either
[task 2018-03-20T20:21:15.165Z] 20:21:15     INFO -  a bug in the called code itself or in the way that mach is calling it.
[task 2018-03-20T20:21:15.165Z] 20:21:15     INFO -  You should consider filing a bug for this issue.
[task 2018-03-20T20:21:15.165Z] 20:21:15     INFO -  If filing a bug, please include the full output of mach, including this error
[task 2018-03-20T20:21:15.165Z] 20:21:15     INFO -  message.
[task 2018-03-20T20:21:15.165Z] 20:21:15     INFO -  The details of the failure are as follows:
[task 2018-03-20T20:21:15.166Z] 20:21:15     INFO -  ConnectionError: HTTPSConnectionPool(host='cloud-mirror-production-us-east-1.s3.amazonaws.com', port=443): Max retries exceeded with url: /https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FT0sgTTO4RF2TUtP1hHMneg%2F0%2Fpublic%2FchainOfTrust.json.asc (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f41ee6041d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
[task 2018-03-20T20:21:15.166Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/mach_commands.py", line 1426, in artifact_toolchain
[task 2018-03-20T20:21:15.166Z] 20:21:15     INFO -      record = ArtifactRecord(task_id, name)
[task 2018-03-20T20:21:15.166Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/mach_commands.py", line 1321, in __init__
[task 2018-03-20T20:21:15.166Z] 20:21:15     INFO -      get_artifact_url(task_id, 'public/chainOfTrust.json.asc'))
[task 2018-03-20T20:21:15.166Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 480, in get
[task 2018-03-20T20:21:15.166Z] 20:21:15     INFO -      return self.request('GET', url, **kwargs)
[task 2018-03-20T20:21:15.167Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 468, in request
[task 2018-03-20T20:21:15.167Z] 20:21:15     INFO -      resp = self.send(prep, **send_kwargs)
[task 2018-03-20T20:21:15.167Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 597, in send
[task 2018-03-20T20:21:15.167Z] 20:21:15     INFO -      history = [resp for resp in gen] if allow_redirects else []
[task 2018-03-20T20:21:15.167Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 195, in resolve_redirects
[task 2018-03-20T20:21:15.167Z] 20:21:15     INFO -      **adapter_kwargs
[task 2018-03-20T20:21:15.168Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/third_party/python/requests/requests/sessions.py", line 576, in send
[task 2018-03-20T20:21:15.168Z] 20:21:15     INFO -      r = adapter.send(request, **kwargs)
[task 2018-03-20T20:21:15.168Z] 20:21:15     INFO -    File "/builds/worker/workspace/build/src/third_party/python/requests/requests/adapters.py", line 437, in send
[task 2018-03-20T20:21:15.168Z] 20:21:15     INFO -      raise ConnectionError(e, request=request)
[task 2018-03-20T20:21:15.178Z] 20:21:15    ERROR - Return code: 1
[task 2018-03-20T20:21:15.179Z] 20:21:15    ERROR - 1 not in success codes: [0]
[task 2018-03-20T20:21:15.179Z] 20:21:15  WARNING - setting return code to 2
[task 2018-03-20T20:21:15.179Z] 20:21:15    FATAL - Halting on failure while running ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/build/src/mach', 'artifact', 'toolchain', '-v', '--retry', '4', '--artifact-manifest', '/builds/worker/workspace/build/src/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache', 'public/build/clang.tar.xz@T0sgTTO4RF2TUtP1hHMneg', 'public/build/rustc.tar.xz@aHJQvxAuQ4ynvdPwaqWIMw', 'public/build/sccache2.tar.xz@EOWg-A0hTCq_sr7aPSMx7g']
[task 2018-03-20T20:21:15.179Z] 20:21:15    FATAL - Running post_fatal callback...
[task 2018-03-20T20:21:15.179Z] 20:21:15    FATAL - Exiting 2
[task 2018-03-20T20:21:15.179Z] 20:21:15     INFO - [mozharness: 2018-03-20 20:21:15.179689Z] Finished build step (failed)
[task 2018-03-20T20:21:15.179Z] 20:21:15     INFO - Running post-run listener: _summarize
[task 2018-03-20T20:21:15.179Z] 20:21:15    ERROR - # TBPL FAILURE #
[task 2018-03-20T20:21:15.180Z] 20:21:15     INFO - [mozharness: 2018-03-20 20:21:15.180093Z] FxDesktopBuild summary:
[task 2018-03-20T20:21:15.180Z] 20:21:15    ERROR - # TBPL FAILURE #

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

cloud-mirror-production-us-east-1 is an AWS system used by TC for serving artifacts, and the job is running on AWS too. Do we have any option other than increasing the number of retries ? The call to mach artifact is ~14 minutes before the error messages so there must already be considerable backoff configured.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

It's not a system, just an S3 bucket, and in the same region as the EC2 instance in question.  So, basically, yes, increasing the number of retries is the only option here.

Comment 3

[Intermittent Failures Robot]

2 failures in 822 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* windows2012-32: 1
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447584&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 5

[Intermittent Failures Robot]

1 failures in 718 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows2012-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447584&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 6

[Intermittent Failures Robot]

1 failures in 3910 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows2012-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447584&startday=2019-03-25&endday=2019-03-31&tree=all