Intermittent wpt internal server errors (500) trying to download smybols after crash

NEW
Unassigned

Status

Testing
General
--
major
9 months ago
9 months ago

People

(Reporter: aryx, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

E.g. https://treeherder.mozilla.org/logviewer.html#?job_id=116605942&repo=autoland

[task 2017-07-22T07:02:34.534853Z] 07:02:34     INFO - TEST-START | /content-security-policy/frame-src/frame-src-self-unique-origin.html
[task 2017-07-22T07:02:34.697152Z] 07:02:34     INFO - PID 2372 | ExceptionHandler::GenerateDump cloned child 2441
[task 2017-07-22T07:02:34.697711Z] 07:02:34     INFO - PID 2372 | ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[task 2017-07-22T07:02:34.698187Z] 07:02:34     INFO - PID 2372 | ExceptionHandler::WaitForContinueSignal waiting for continue signal...
[task 2017-07-22T07:02:34.916185Z] 07:02:34     INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/cZS1rmCrRBuEVsa0ifJ-UQ/artifacts/public/build/target.crashreporter-symbols.zip
[task 2017-07-22T07:02:35.315164Z] 07:02:35    ERROR - Traceback (most recent call last):
[task 2017-07-22T07:02:35.316268Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 347, in run
[task 2017-07-22T07:02:35.316976Z] 07:02:35    ERROR -     new_state = self.wait_event()
[task 2017-07-22T07:02:35.317740Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 421, in wait_event
[task 2017-07-22T07:02:35.318320Z] 07:02:35    ERROR -     return f(*data)
[task 2017-07-22T07:02:35.321191Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 556, in test_ended
[task 2017-07-22T07:02:35.321254Z] 07:02:35    ERROR -     self.browser.log_crash(test.id)
[task 2017-07-22T07:02:35.321333Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 227, in log_crash
[task 2017-07-22T07:02:35.321395Z] 07:02:35    ERROR -     self.browser.log_crash(process=self.browser_pid, test=test_id)
[task 2017-07-22T07:02:35.321480Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 321, in log_crash
[task 2017-07-22T07:02:35.322033Z] 07:02:35    ERROR -     test=test)
[task 2017-07-22T07:02:35.322772Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 126, in log_crashes
[task 2017-07-22T07:02:35.323225Z] 07:02:35    ERROR -     stackwalk_binary=stackwalk_binary):
[task 2017-07-22T07:02:35.323775Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 212, in __iter__
[task 2017-07-22T07:02:35.324290Z] 07:02:35    ERROR -     rv = self._process_dump_file(path, extra)
[task 2017-07-22T07:02:35.324905Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 234, in _process_dump_file
[task 2017-07-22T07:02:35.325307Z] 07:02:35    ERROR -     self._get_symbols()
[task 2017-07-22T07:02:35.325902Z] 07:02:35    ERROR -   File "/home/worker/workspace/build/venv/local/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 180, in _get_symbols
[task 2017-07-22T07:02:35.326390Z] 07:02:35    ERROR -     data = urllib2.urlopen(self.symbols_path)
[task 2017-07-22T07:02:35.326911Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
[task 2017-07-22T07:02:35.327345Z] 07:02:35    ERROR -     return opener.open(url, data, timeout)
[task 2017-07-22T07:02:35.327902Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 435, in open
[task 2017-07-22T07:02:35.328441Z] 07:02:35    ERROR -     response = meth(req, response)
[task 2017-07-22T07:02:35.329007Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
[task 2017-07-22T07:02:35.329443Z] 07:02:35    ERROR -     'http', request, response, code, msg, hdrs)
[task 2017-07-22T07:02:35.329877Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 473, in error
[task 2017-07-22T07:02:35.330465Z] 07:02:35    ERROR -     return self._call_chain(*args)
[task 2017-07-22T07:02:35.330862Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
[task 2017-07-22T07:02:35.331257Z] 07:02:35    ERROR -     result = func(*args)
[task 2017-07-22T07:02:35.331865Z] 07:02:35    ERROR -   File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
[task 2017-07-22T07:02:35.332316Z] 07:02:35    ERROR -     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
[task 2017-07-22T07:02:35.332905Z] 07:02:35    ERROR - HTTPError: HTTP Error 500: Internal Server Error

Comment 1

9 months ago
23 failures in 48 pushes (0.479 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 14
* mozilla-inbound: 9

Platform breakdown:
* windows10-64: 8
* windows7-32: 7
* linux64-stylo: 3
* windows8-64: 2
* linux32: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1383464&startday=2017-07-22&endday=2017-07-22&tree=all

Comment 2

9 months ago
25 failures in 822 pushes (0.03 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 16
* mozilla-inbound: 9

Platform breakdown:
* windows10-64: 9
* windows7-32: 7
* linux64-stylo: 4
* windows8-64: 2
* linux32: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1383464&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 3

9 months ago
Two ways we can improve this:

1) have mozcrash retry downloads, and only report failure after giving up on retries

2) reduce number of 500 errors returned by queue.tc.net.
Component: Buildduty → General
Product: Release Engineering → Testing
QA Contact: catlee
You need to log in before you can comment on or make changes to this bug.