1277506 - store-error-lines should not retry/raise for HTTP 403 and 404 responses

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

Description

[Ed Morley [:emorley]]

For taskcluster log URLs, it's unfortunately expected that they can sometimes 404 or 403 - see bug 1154248.

As such, the log parser special-cases these, so:
(a) it doesn't keep retrying them (if the log is 404 it's not going to be there on the next retry)
(b) it doesn't raise, reducing noise on New Relic

See bug 1155647, bug 1159934.

However store-error-lines doesn't do this, so is currently generating a lot of retries for 403 cases:
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/556195-c243ea31-28a5-11e6-b947-b82a72d22a14/similar_errors

See the log parser implementation:
https://github.com/mozilla/treeherder/blob/3402f00fbc99d336118f5cde0fe3c36ec8f11b76/treeherder/log_parser/utils.py#L67-L89

Note also that it avoids the retry (but still raises) for non-HTTPError exceptions (eg a parsing error), since we still want the New Relic entry, but there's no point continually trying to parse the same log content, if it causes an exception. 

Perhaps we should use a common log downloader function between the log parser and store-error-lines? It seems like store-error-lines should also be streaming the content, like the log parser does? (Note the log parser currently uses urllib2, but bug 1165356 is wanting to switch it to requests.)

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:store_error_summary_ignore_404 > mozilla:master

Comment 2

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/d01101a0343ef94b5a4d332b593f626865fa0c6a
Bug 1277506 - Don't raise an error when storing failure lines generates 403/404.

This matches the behaviour of fetching the full logs and represents a
belief that 404 errors will not magically correct themselves when we
retry.

https://github.com/mozilla/treeherder/commit/27fe7e730f7c6f59971018c840f0e510b0bb5083
Merge pull request #1554 from mozilla/store_error_summary_ignore_404

Bug 1277506 - Don't raise an error when storing failure lines generates 403/404.