Closed Bug 1383464 Opened 7 years ago Closed 1 year ago

Intermittent wpt internal server errors (500) trying to download symbols after crash

Categories

(Testing :: General, defect, P3)

Product:
Testing
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: aryx, Unassigned)

Details

(Keywords: intermittent-failure) 

E.g. https://treeherder.mozilla.org/logviewer.html#?job_id=116605942&repo=autoland

[task 2017-07-22T07:02:34.534853Z] 07:02:34     INFO - TEST-START | /content-security-policy/frame-src/frame-src-self-unique-origin.html
[task 2017-07-22T07:02:34.697152Z] 07:02:34     INFO - PID 2372 | ExceptionHandler::GenerateDump cloned child 2441
[task 2017-07-22T07:02:34.697711Z] 07:02:34     INFO - PID 2372 | ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[task 2017-07-22T07:02:34.698187Z] 07:02:34     INFO - PID 2372 | ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[task 2017-07-22T07:02:34.916185Z] 07:02:34     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/cZS1rmCrRBuEVsa0ifJ-UQ/artifacts/public/build/target.crashreporter-symbols.zip
[task 2017-07-22T07:02:35.315164Z] 07:02:35    ERROR - Traceback (most recent call last):
[task 2017-07-22T07:02:35.316268Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 347, in run
[task 2017-07-22T07:02:35.316976Z] 07:02:35    ERROR -     new_state = self.wait_event()
[task 2017-07-22T07:02:35.317740Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 421, in wait_event
[task 2017-07-22T07:02:35.318320Z] 07:02:35    ERROR -     return f(*data)
[task 2017-07-22T07:02:35.321191Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 556, in test_ended
[task 2017-07-22T07:02:35.321254Z] 07:02:35    ERROR -     self.browser.log_crash(test.id)
[task 2017-07-22T07:02:35.321333Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 227, in log_crash
[task 2017-07-22T07:02:35.321395Z] 07:02:35    ERROR -     self.browser.log_crash(process=self.browser_pid, test=test_id)
[task 2017-07-22T07:02:35.321480Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 321, in log_crash
[task 2017-07-22T07:02:35.322033Z] 07:02:35    ERROR -     test=test)
[task 2017-07-22T07:02:35.322772Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 126, in log_crashes
[task 2017-07-22T07:02:35.323225Z] 07:02:35    ERROR -     stackwalk_binary=stackwalk_binary):
[task 2017-07-22T07:02:35.323775Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 212, in __iter__
[task 2017-07-22T07:02:35.324290Z] 07:02:35    ERROR -     rv = self._process_dump_file(path, extra)
[task 2017-07-22T07:02:35.324905Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 234, in _process_dump_file
[task 2017-07-22T07:02:35.325307Z] 07:02:35    ERROR -     self._get_symbols()
[task 2017-07-22T07:02:35.325902Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 180, in _get_symbols
[task 2017-07-22T07:02:35.326390Z] 07:02:35    ERROR -     data = urllib2.urlopen(self.symbols_path)
[task 2017-07-22T07:02:35.326911Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
[task 2017-07-22T07:02:35.327345Z] 07:02:35    ERROR -     return opener.open(url, data, timeout)
[task 2017-07-22T07:02:35.327902Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 435, in open
[task 2017-07-22T07:02:35.328441Z] 07:02:35    ERROR -     response = meth(req, response)
[task 2017-07-22T07:02:35.329007Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
[task 2017-07-22T07:02:35.329443Z] 07:02:35    ERROR -     'http', request, response, code, msg, hdrs)
[task 2017-07-22T07:02:35.329877Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 473, in error
[task 2017-07-22T07:02:35.330465Z] 07:02:35    ERROR -     return self._call_chain(*args)
[task 2017-07-22T07:02:35.330862Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
[task 2017-07-22T07:02:35.331257Z] 07:02:35    ERROR -     result = func(*args)
[task 2017-07-22T07:02:35.331865Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
[task 2017-07-22T07:02:35.332316Z] 07:02:35    ERROR -     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
[task 2017-07-22T07:02:35.332905Z] 07:02:35    ERROR - HTTPError: HTTP Error 500: Internal Server Error
Two ways we can improve this:

1) have mozcrash retry downloads, and only report failure after giving up on retries

2) reduce number of 500 errors returned by queue.tc.net.
Component: Buildduty → General
Product: Release Engineering → Testing
QA Contact: catlee
Priority: -- → P3
Summary: Intermittent wpt internal server errors (500) trying to download smybols after crash → Intermittent wpt internal server errors (500) trying to download symbols after crash
Severity: major → S2

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.