Closed Bug 1277506 Opened 6 years ago Closed 6 years ago

store-error-lines should not retry/raise for HTTP 403 and 404 responses

Categories

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Assigned: jgraham)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

For taskcluster log URLs, it's unfortunately expected that they can sometimes 404 or 403 - see bug 1154248.

As such, the log parser special-cases these, so:
(a) it doesn't keep retrying them (if the log is 404 it's not going to be there on the next retry)
(b) it doesn't raise, reducing noise on New Relic

See bug 1155647, bug 1159934.

However store-error-lines doesn't do this, so is currently generating a lot of retries for 403 cases:
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/556195-c243ea31-28a5-11e6-b947-b82a72d22a14/similar_errors

See the log parser implementation:
https://github.com/mozilla/treeherder/blob/3402f00fbc99d336118f5cde0fe3c36ec8f11b76/treeherder/log_parser/utils.py#L67-L89

Note also that it avoids the retry (but still raises) for non-HTTPError exceptions (eg a parsing error), since we still want the New Relic entry, but there's no point continually trying to parse the same log content, if it causes an exception. 

Perhaps we should use a common log downloader function between the log parser and store-error-lines? It seems like store-error-lines should also be streaming the content, like the log parser does? (Note the log parser currently uses urllib2, but bug 1165356 is wanting to switch it to requests.)
Flags: needinfo?(james)
Flags: needinfo?(james)
Attachment #8759200 - Flags: review?(emorley)
Attachment #8759200 - Flags: review?(emorley) → review+
Assignee: nobody → james
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/d01101a0343ef94b5a4d332b593f626865fa0c6a
Bug 1277506 - Don't raise an error when storing failure lines generates 403/404.

This matches the behaviour of fetching the full logs and represents a
belief that 404 errors will not magically correct themselves when we
retry.

https://github.com/mozilla/treeherder/commit/27fe7e730f7c6f59971018c840f0e510b0bb5083
Merge pull request #1554 from mozilla/store_error_summary_ignore_404

Bug 1277506 - Don't raise an error when storing failure lines generates 403/404.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.