Closed Bug 1383464 Opened 7 years ago Closed 2 years ago

Intermittent wpt internal server errors (500) trying to download symbols after crash

Categories

(Testing :: General, defect, P3)

Product:
Testing
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: aryx, Unassigned)

Details

(Keywords: intermittent-failure)

E.g. https://treeherder.mozilla.org/logviewer.html#?job_id=116605942&repo=autoland [task 2017-07-22T07:02:34.534853Z] 07:02:34 INFO - TEST-START | /content-security-policy/frame-src/frame-src-self-unique-origin.html [task 2017-07-22T07:02:34.697152Z] 07:02:34 INFO - PID 2372 | ExceptionHandler::GenerateDump cloned child 2441 [task 2017-07-22T07:02:34.697711Z] 07:02:34 INFO - PID 2372 | ExceptionHandler::SendContinueSignalToChild sent continue signal to child [task 2017-07-22T07:02:34.698187Z] 07:02:34 INFO - PID 2372 | ExceptionHandler::WaitForContinueSignal waiting for continue signal... [task 2017-07-22T07:02:34.916185Z] 07:02:34 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/cZS1rmCrRBuEVsa0ifJ-UQ/artifacts/public/build/target.crashreporter-symbols.zip [task 2017-07-22T07:02:35.315164Z] 07:02:35 ERROR - Traceback (most recent call last): [task 2017-07-22T07:02:35.316268Z] 07:02:35 ERROR - File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 347, in run [task 2017-07-22T07:02:35.316976Z] 07:02:35 ERROR - new_state = self.wait_event() [task 2017-07-22T07:02:35.317740Z] 07:02:35 ERROR - File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 421, in wait_event [task 2017-07-22T07:02:35.318320Z] 07:02:35 ERROR - return f(*data) [task 2017-07-22T07:02:35.321191Z] 07:02:35 ERROR - File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 556, in test_ended [task 2017-07-22T07:02:35.321254Z] 07:02:35 ERROR - self.browser.log_crash(test.id) [task 2017-07-22T07:02:35.321333Z] 07:02:35 ERROR - File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 227, in log_crash [task 2017-07-22T07:02:35.321395Z] 07:02:35 ERROR - self.browser.log_crash(process=self.browser_pid, test=test_id) [task 2017-07-22T07:02:35.321480Z] 07:02:35 ERROR - File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 321, in log_crash [task 2017-07-22T07:02:35.322033Z] 07:02:35 ERROR - test=test) [task 2017-07-22T07:02:35.322772Z] 07:02:35 ERROR - File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 126, in log_crashes [task 2017-07-22T07:02:35.323225Z] 07:02:35 ERROR - stackwalk_binary=stackwalk_binary): [task 2017-07-22T07:02:35.323775Z] 07:02:35 ERROR - File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 212, in __iter__ [task 2017-07-22T07:02:35.324290Z] 07:02:35 ERROR - rv = self._process_dump_file(path, extra) [task 2017-07-22T07:02:35.324905Z] 07:02:35 ERROR - File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 234, in _process_dump_file [task 2017-07-22T07:02:35.325307Z] 07:02:35 ERROR - self._get_symbols() [task 2017-07-22T07:02:35.325902Z] 07:02:35 ERROR - File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 180, in _get_symbols [task 2017-07-22T07:02:35.326390Z] 07:02:35 ERROR - data = urllib2.urlopen(self.symbols_path) [task 2017-07-22T07:02:35.326911Z] 07:02:35 ERROR - File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen [task 2017-07-22T07:02:35.327345Z] 07:02:35 ERROR - return opener.open(url, data, timeout) [task 2017-07-22T07:02:35.327902Z] 07:02:35 ERROR - File "/usr/lib/python2.7/urllib2.py", line 435, in open [task 2017-07-22T07:02:35.328441Z] 07:02:35 ERROR - response = meth(req, response) [task 2017-07-22T07:02:35.329007Z] 07:02:35 ERROR - File "/usr/lib/python2.7/urllib2.py", line 548, in http_response [task 2017-07-22T07:02:35.329443Z] 07:02:35 ERROR - 'http', request, response, code, msg, hdrs) [task 2017-07-22T07:02:35.329877Z] 07:02:35 ERROR - File "/usr/lib/python2.7/urllib2.py", line 473, in error [task 2017-07-22T07:02:35.330465Z] 07:02:35 ERROR - return self._call_chain(*args) [task 2017-07-22T07:02:35.330862Z] 07:02:35 ERROR - File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain [task 2017-07-22T07:02:35.331257Z] 07:02:35 ERROR - result = func(*args) [task 2017-07-22T07:02:35.331865Z] 07:02:35 ERROR - File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default [task 2017-07-22T07:02:35.332316Z] 07:02:35 ERROR - raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) [task 2017-07-22T07:02:35.332905Z] 07:02:35 ERROR - HTTPError: HTTP Error 500: Internal Server Error
Two ways we can improve this: 1) have mozcrash retry downloads, and only report failure after giving up on retries 2) reduce number of 500 errors returned by queue.tc.net.
Component: Buildduty → General
Product: Release Engineering → Testing
QA Contact: catlee
Priority: -- → P3
Summary: Intermittent wpt internal server errors (500) trying to download smybols after crash → Intermittent wpt internal server errors (500) trying to download symbols after crash
Severity: major → S2

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.