1454858 - Intermittent HTTPError: HTTP Error 503: Service Unavailable

Status: RESOLVED INVALID

(Taskcluster :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=174243539&repo=autoland

https://queue.taskcluster.net/v1/task/S2eirkEXSPOt9w6_OuVRHw/runs/0/artifacts/public/logs/live_backing.log

[task 2018-04-18T01:52:52.060Z] 01:52:52     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/Xsa9rMiwRMe61DLixR-aLA/artifacts/public/build/target.crashreporter-symbols.zip
[task 2018-04-18T01:52:52.077Z] 01:52:52     INFO - PID 16548 | --DOCSHELL 0x7ffb00173000 == 4 [pid = 16548] [id = {9f15479b-70a1-4429-9ff6-1cf9bdfbe13e}]
[task 2018-04-18T01:52:52.194Z] 01:52:52     INFO - PID 16548 | [Parent 16548, Main Thread] WARNING: unable to post continuation event: file /builds/worker/workspace/build/src/xpcom/io/nsStreamUtils.cpp, line 490
[task 2018-04-18T01:52:52.579Z] 01:52:52    ERROR - Traceback (most recent call last):
[task 2018-04-18T01:52:52.579Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 360, in run
[task 2018-04-18T01:52:52.580Z] 01:52:52    ERROR -     new_state = self.wait_event()
[task 2018-04-18T01:52:52.580Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 435, in wait_event
[task 2018-04-18T01:52:52.581Z] 01:52:52    ERROR -     return f(*data)
[task 2018-04-18T01:52:52.581Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 579, in test_ended
[task 2018-04-18T01:52:52.582Z] 01:52:52    ERROR -     self.browser.log_crash(test.id)
[task 2018-04-18T01:52:52.582Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 237, in log_crash
[task 2018-04-18T01:52:52.582Z] 01:52:52    ERROR -     self.browser.log_crash(process=self.browser_pid, test=test_id)
[task 2018-04-18T01:52:52.583Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 344, in log_crash
[task 2018-04-18T01:52:52.583Z] 01:52:52    ERROR -     test=test)
[task 2018-04-18T01:52:52.584Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 128, in log_crashes
[task 2018-04-18T01:52:52.584Z] 01:52:52    ERROR -     stackwalk_binary=stackwalk_binary):
[task 2018-04-18T01:52:52.584Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 214, in __iter__
[task 2018-04-18T01:52:52.585Z] 01:52:52    ERROR -     rv = self._process_dump_file(path, extra)
[task 2018-04-18T01:52:52.585Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 236, in _process_dump_file
[task 2018-04-18T01:52:52.586Z] 01:52:52    ERROR -     self._get_symbols()
[task 2018-04-18T01:52:52.586Z] 01:52:52    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 182, in _get_symbols
[task 2018-04-18T01:52:52.586Z] 01:52:52    ERROR -     data = urllib2.urlopen(self.symbols_path)
[task 2018-04-18T01:52:52.587Z] 01:52:52    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
[task 2018-04-18T01:52:52.587Z] 01:52:52    ERROR -     return opener.open(url, data, timeout)
[task 2018-04-18T01:52:52.588Z] 01:52:52    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 435, in open
[task 2018-04-18T01:52:52.588Z] 01:52:52    ERROR -     response = meth(req, response)
[task 2018-04-18T01:52:52.588Z] 01:52:52    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
[task 2018-04-18T01:52:52.589Z] 01:52:52    ERROR -     'http', request, response, code, msg, hdrs)
[task 2018-04-18T01:52:52.589Z] 01:52:52    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 473, in error
[task 2018-04-18T01:52:52.590Z] 01:52:52    ERROR -     return self._call_chain(*args)
[task 2018-04-18T01:52:52.590Z] 01:52:52    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
[task 2018-04-18T01:52:52.590Z] 01:52:52    ERROR -     result = func(*args)
[task 2018-04-18T01:52:52.591Z] 01:52:52    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
[task 2018-04-18T01:52:52.591Z] 01:52:52    ERROR -     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
[task 2018-04-18T01:52:52.592Z] 01:52:52    ERROR - HTTPError: HTTP Error 503: Service Unavailable
[task 2018-04-18T01:52:52.592Z] 01:52:52    ERROR - 
[task 2018-04-18T01:52:52.616Z] 01:52:52     INFO - Browser exited with return code -15

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

503's typically come from Amazon S3.  S3 provides five+ nines of assurance that they won't lose your data, but only one or two nines of assurance that they will serve it to you on the first try.  So this request should retry.  Basically any use of raw `urllib2` in Python is destined for sadness -- python-requests is much more robust.

Comment 2

[Intermittent Failures Robot]

1 failures in 782 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1454858&startday=2018-04-16&endday=2018-04-22&tree=trunk

Comment 3

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Comment 4

[Andreea Pavel [:apavel]]

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=180641684&repo=autoland&lineNumber=99

[taskcluster 2018-05-29T03:14:02.326Z]    Exit Code: 0
[taskcluster 2018-05-29T03:14:02.326Z]    User Time: 0s
[taskcluster 2018-05-29T03:14:02.326Z]  Kernel Time: 31.25ms
[taskcluster 2018-05-29T03:14:02.326Z]    Wall Time: 50.0008ms
[taskcluster 2018-05-29T03:14:02.326Z]       Result: SUCCEEDED
[taskcluster 2018-05-29T03:14:02.326Z] Executing command 4: "c:\Program Files\Mercurial\hg.exe" robustcheckout --sharebase y:\hg-shared --purge --upstream https://hg.mozilla.org/mozilla-unified --revision f79ea1a6633dc7a3716b8a9f4716fc7ba3fe0e2e https://hg.mozilla.org/integration/autoland .\build\src

z:\build>"c:\Program Files\Mercurial\hg.exe" robustcheckout --sharebase y:\hg-shared --purge --upstream https://hg.mozilla.org/mozilla-unified --revision f79ea1a6633dc7a3716b8a9f4716fc7ba3fe0e2e https://hg.mozilla.org/integration/autoland .\build\src 
(using Mercurial 4.5.3)
ensuring https://hg.mozilla.org/integration/autoland@f79ea1a6633dc7a3716b8a9f4716fc7ba3fe0e2e is available at .\build\src
(cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
unhandled URLError; reason type: str; value: Service UnavailablePERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "clone_errored", "shouldAlert": false, "subtests": [], "value": 0.031000137329101562}, {"extraOptions": ["c4.4xlarge"], "lowerIsBetter": true, "name": "overall", "shouldAlert": false, "subtests": [], "value": 0.8310000896453857}]}
Traceback (most recent call last):
  File "mercurial\scmutil.pyc", line 154, in callcatch
  File "mercurial\dispatch.pyc", line 314, in _runcatchfunc
  File "mercurial\dispatch.pyc", line 918, in _dispatch
  File "mercurial\dispatch.pyc", line 673, in runcommand
  File "mercurial\dispatch.pyc", line 926, in _runcommand
  File "mercurial\dispatch.pyc", line 915, in <lambda>
  File "mercurial\util.pyc", line 1195, in check
  File "C:/mozilla-build/robustcheckout.py", line 260, in robustcheckout
  File "C:/mozilla-build/robustcheckout.py", line 542, in _docheckout
  File "mercurial\hg.pyc", line 548, in clone
  File "mercurial\peer.pyc", line 97, in plain
  File "mercurial\wireproto.pyc", line 451, in _submitone
  File "mercurial\httppeer.pyc", line 407, in _call
  File "mercurial\httppeer.pyc", line 336, in _callstream
  File "mercurial\httppeer.pyc", line 193, in _openurl
  File "urllib2.pyc", line 435, in open
  File "urllib2.pyc", line 548, in http_response
  File "urllib2.pyc", line 473, in error
  File "urllib2.pyc", line 407, in _call_chain
  File "urllib2.pyc", line 556, in http_error_default
HTTPError: HTTP Error 503: Service Unavailable
abort: HTTP Error 503: Service Unavailable
[taskcluster 2018-05-29T03:14:03.549Z]    Exit Code: 255
[taskcluster 2018-05-29T03:14:03.549Z]    User Time: 0s
[taskcluster 2018-05-29T03:14:03.549Z]  Kernel Time: 15.625ms
[taskcluster 2018-05-29T03:14:03.549Z]    Wall Time: 1.221999s
[taskcluster 2018-05-29T03:14:03.549Z]       Result: FAILED
[taskcluster 2018-05-29T03:14:03.549Z] === Task Finished ===
[taskcluster 2018-05-29T03:14:03.549Z] Task Duration: 1.4939972s
[taskcluster:error] Uploading error artifact public/build from file public/build with message "Could not read directory 'Z:\\task_1527561042\\public\\build'", reason "file-missing-on-worker" and expiry 2019-05-29T03:12:56.273Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'Z:\task_1527561042\public\build'
[taskcluster 2018-05-29T03:14:04.269Z] Uploading artifact public/logs/certified.log from file generic-worker\certified.log with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-05-29T03:12:56.273Z
[taskcluster 2018-05-29T03:14:05.185Z] Uploading artifact public/chainOfTrust.json.asc from file generic-worker\chainOfTrust.json.asc with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-05-29T03:12:56.273Z
[taskcluster 2018-05-29T03:14:05.751Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/IMir8CHARLi0n4zTK31X2A/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-05-29T03:12:56.273Z
[taskcluster:error] exit status 255
[taskcluster:error] file-missing-on-worker: Could not read directory 'Z:\task_1527561042\public\build'

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

This is a different error, and one in hg.mozilla.org and/or robustcheckout's handling of such errors -- not to do with taskcluster.

Comment 6

[Andreea Pavel [:apavel]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #5)
> This is a different error, and one in hg.mozilla.org and/or robustcheckout's
> handling of such errors -- not to do with taskcluster.

:dustin ok, thank you. 
Aryx: do i file a new bug for the above log? https://bugzilla.mozilla.org/show_bug.cgi?id=1454858#c4

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Yes, new bug please. Developer Services :: hg.mozilla.org.

Comment 8

[Intermittent Failures Robot]

1 failures in 655 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows2012-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1454858&startday=2018-05-28&endday=2018-06-03&tree=trunk

Comment 9

[Natalia Csoregi [:nataliaCs]]

https://treeherder.mozilla.org/logviewer.html#?job_id=195362062&repo=mozilla-inbound&lineNumber=21911

Comment 10

[Intermittent Failures Robot]

3 failures in 2568 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1
* mozilla-inbound: 2

Platform breakdown:
* linux64-asan-reporter: 1
* osx-cross: 1
* osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1454858&startday=2018-08-20&endday=2018-08-26&tree=all

Comment 11

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Comment 12

[Stefan Hindli [:stefan_hindli]]

https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&fromchange=adcf05cc20255510741620dabdbf442a1d3ed0b1&selectedJob=217651594

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

https://taskcluster-artifacts.net/cIA_RQZ1RuSCFdrrj9HDoA/0/public/logs/live_backing.log is not a 503 error..

Comment 14

[Intermittent Failures Robot]

1 failures in 2648 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1454858&startday=2018-12-17&endday=2018-12-23&tree=all

Comment 15

[Cristian Brindusan [:cbrindusan]]

Recent log failure:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221850326&repo=mozilla-central&lineNumber=992
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221850425&repo=mozilla-central&lineNumber=919

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Those are bug 1514431.

Comment 17

[Intermittent Failures Robot]

5 failures in 2378 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 4

Platform breakdown:

• windows10-64-ccov: 2
• linux64-ccov: 2
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1454858&startday=2019-01-14&endday=2019-01-20&tree=all

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=224352144&repo=mozilla-central&lineNumber=1306

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Let's please not make this a catch-all bug for every 503 error in the world. This one, for example, was from relengapi.

Comment 20

[Intermittent Failures Robot]

8 failures in 2383 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 7

Platform breakdown:

• windows10-64: 1
• windows7-32: 1
• windows10-64-ccov: 1
• linux64: 1
• windows7-32-devedition: 1
• linux64-qr: 1
• android-em-4-3-armv7-api16: 1
• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1454858&startday=2019-01-21&endday=2019-01-27&tree=all

Comment 21

[Intermittent Failures Robot]

1 failures in 2628 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• linux64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1454858&startday=2019-01-28&endday=2019-02-03&tree=all