Closed Bug 1523324 Opened 10 months ago Closed 6 months ago

Intermittent requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://queue.taskcluster.net/v1/task/U_XYOXu8SlqXHpJX-1pgyQ/artifacts/public/build/target.generated-files.tar.gz

Categories

(Firefox Build System :: General, defect, P5)

defect

Tracking

(firefox-esr60 fixed, firefox67 fixed, firefox68 fixed)

RESOLVED FIXED
mozilla68
Tracking Status
firefox-esr60 --- fixed
firefox67 --- fixed
firefox68 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: mshal, Mentored)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

#[markdown(off)]
Filed by: nerli [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=224427646&repo=try

https://queue.taskcluster.net/v1/task/ZOdZ6jSoRmK7w0vrRcqRAA/runs/0/artifacts/public/logs/live_backing.log

[task 2019-01-28T11:43:27.436Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-01-28T11:43:28.412Z] Collecting boto3==1.4.4
[task 2019-01-28T11:43:28.731Z] Downloading https://files.pythonhosted.org/packages/91/60/649da03299624f524c8d0cd4c6c73c194023e85dd4938f1e7712ab6888bf/boto3-1.4.4-py2.py3-none-any.whl (127kB)
[task 2019-01-28T11:43:28.869Z] Collecting botocore<1.6.0,>=1.5.0 (from boto3==1.4.4)
[task 2019-01-28T11:43:29.435Z] Downloading https://files.pythonhosted.org/packages/be/83/6807ca70447bf7edf103e0b16553e729a5b7dc8e67c882f1ed57fdc3121c/botocore-1.5.95-py2.py3-none-any.whl (3.6MB)
[task 2019-01-28T11:43:30.073Z] Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.4.4)
[task 2019-01-28T11:43:30.111Z] Downloading https://files.pythonhosted.org/packages/b7/31/05c8d001f7f87f0f07289a5fc0fc3832e9a57f2dbd4d3b0fee70e0d51365/jmespath-0.9.3-py2.py3-none-any.whl
[task 2019-01-28T11:43:30.225Z] Collecting s3transfer<0.2.0,>=0.1.10 (from boto3==1.4.4)
[task 2019-01-28T11:43:30.264Z] Downloading https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl (59kB)
[task 2019-01-28T11:43:30.387Z] Collecting docutils>=0.10 (from botocore<1.6.0,>=1.5.0->boto3==1.4.4)
[task 2019-01-28T11:43:30.423Z] Downloading https://files.pythonhosted.org/packages/50/09/c53398e0005b11f7ffb27b7aa720c617aba53be4fb4f4f3f06b9b5c60f28/docutils-0.14-py2-none-any.whl (543kB)
[task 2019-01-28T11:43:30.618Z] Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.6.0,>=1.5.0->boto3==1.4.4)
[task 2019-01-28T11:43:30.671Z] Downloading https://files.pythonhosted.org/packages/74/68/d87d9b36af36f44254a8d512cbfc48369103a3b9e474be9bdfe536abfc45/python_dateutil-2.7.5-py2.py3-none-any.whl (225kB)
[task 2019-01-28T11:43:30.759Z] Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in ./third_party/python/futures (from s3transfer<0.2.0,>=0.1.10->boto3==1.4.4)
[task 2019-01-28T11:43:30.813Z] Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.6.0,>=1.5.0->boto3==1.4.4)
[task 2019-01-28T11:43:30.855Z] Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
[task 2019-01-28T11:43:30.913Z] Installing collected packages: jmespath, docutils, six, python-dateutil, botocore, s3transfer, boto3
[task 2019-01-28T11:43:31.815Z] Successfully installed boto3-1.4.4 botocore-1.5.95 docutils-0.14 jmespath-0.9.3 python-dateutil-2.7.5 s3transfer-0.1.13 six-1.12.0
[task 2019-01-28T11:43:31.958Z] You are using pip version 9.0.3, however version 19.0.1 is available.
[task 2019-01-28T11:43:31.958Z] You should consider upgrading via the 'pip install --upgrade pip' command.
[task 2019-01-28T11:43:31.984Z] INFO - MainThread - Using AWS credentials from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-1/gecko-generated-sources-upload"
[task 2019-01-28T11:43:32.154Z] INFO - MainThread - Fetching generated sources artifact: "https://queue.taskcluster.net/v1/task/U_XYOXu8SlqXHpJX-1pgyQ/artifacts/public/build/target.generated-files.tar.gz"
[task 2019-01-28T11:44:04.179Z] INFO - MainThread - Fetch HTTP status: 503, 506 bytes downloaded in 32.026s
[task 2019-01-28T11:44:04.179Z] Traceback (most recent call last):
[task 2019-01-28T11:44:04.179Z] File "build/upload_generated_sources.py", line 161, in <module>
[task 2019-01-28T11:44:04.179Z] sys.exit(main(sys.argv[1:]))
[task 2019-01-28T11:44:04.179Z] File "build/upload_generated_sources.py", line 155, in main
[task 2019-01-28T11:44:04.180Z] do_work(region=region, bucket=bucket, artifact=args.artifact)
[task 2019-01-28T11:44:04.180Z] File "build/upload_generated_sources.py", line 117, in do_work
[task 2019-01-28T11:44:04.180Z] res.raise_for_status()
[task 2019-01-28T11:44:04.180Z] File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 840, in raise_for_status
[task 2019-01-28T11:44:04.180Z] raise HTTPError(http_error_msg, response=self)
[task 2019-01-28T11:44:04.181Z] requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://queue.taskcluster.net/v1/task/U_XYOXu8SlqXHpJX-1pgyQ/artifacts/public/build/target.generated-files.tar.gz
[taskcluster 2019-01-28 11:44:04.568Z] === Task Finished ===
[taskcluster 2019-01-28 11:44:04.568Z] Unsuccessful task run with exit code: 1 completed in 249.96 seconds

Status: NEW → RESOLVED
Closed: 8 months ago
Resolution: --- → INCOMPLETE

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=242835615&repo=autoland&lineNumber=251

[task 2019-04-26T10:37:28.480Z] Installing collected packages: jmespath, docutils, six, python-dateutil, botocore, s3transfer, boto3
[task 2019-04-26T10:37:29.393Z] Successfully installed boto3-1.4.4 botocore-1.5.95 docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 s3transfer-0.1.13 six-1.12.0
[task 2019-04-26T10:37:29.607Z] You are using pip version 9.0.3, however version 19.1 is available.
[task 2019-04-26T10:37:29.607Z] You should consider upgrading via the 'pip install --upgrade pip' command.
[task 2019-04-26T10:37:29.630Z] INFO - MainThread - Using AWS credentials from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-generated-sources-upload"
[task 2019-04-26T10:37:30.114Z] INFO - MainThread - Fetching generated sources artifact: "https://queue.taskcluster.net/v1/task/BKFlZya_Ri6Dsbq2fZgm5w/artifacts/public/build/target.generated-files.tar.gz"
[task 2019-04-26T10:38:01.752Z] INFO - MainThread - Fetch HTTP status: 503, 506 bytes downloaded in 31.639s
[task 2019-04-26T10:38:01.752Z] Traceback (most recent call last):
[task 2019-04-26T10:38:01.753Z] File "build/upload_generated_sources.py", line 161, in <module>
[task 2019-04-26T10:38:01.753Z] sys.exit(main(sys.argv[1:]))
[task 2019-04-26T10:38:01.753Z] File "build/upload_generated_sources.py", line 155, in main
[task 2019-04-26T10:38:01.753Z] do_work(region=region, bucket=bucket, artifact=args.artifact)
[task 2019-04-26T10:38:01.753Z] File "build/upload_generated_sources.py", line 117, in do_work
[task 2019-04-26T10:38:01.753Z] res.raise_for_status()
[task 2019-04-26T10:38:01.753Z] File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/models.py", line 840, in raise_for_status
[task 2019-04-26T10:38:01.753Z] raise HTTPError(http_error_msg, response=self)
[task 2019-04-26T10:38:01.753Z] requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://queue.taskcluster.net/v1/task/BKFlZya_Ri6Dsbq2fZgm5w/artifacts/public/build/target.generated-files.tar.gz
[taskcluster 2019-04-26 10:38:02.123Z] === Task Finished ===

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

This should be retrying that download.

You can see how to configure python-requests to automatically retry in
taskcluster/taskgraph/util/taskcluster.py
and apply that in
build/upload_generated_sources.py

Mentor: dustin
Product: Taskcluster → Firefox Build System

Occasionally the taskcluster fetches can fail, so make sure the requests
library automatically retries if possible.

Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4dabc47dec12
Add retries to upload_generated_sources.py; r=nalexander
Status: REOPENED → RESOLVED
Closed: 8 months ago6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
You need to log in before you can comment on or make changes to this bug.