1078415 - Limit the number of log error lines stored in the job artefact

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

Description

[Ed Morley [:emorley]]

Broken out from a recent email.

In order to reduce DB consumption from job artefacts from runs that contain thousands of failures (eg all green talos tp5 jobs, bug 1078409), we should cap the number of error lines we store in the job artefacts.

If someone needs to look at later errors, they can just use the log viewer (and scroll down more) or look at the raw log.

In addition, we should limit the number of failures shown in the treeherder failure summary tab (and likely to a value lower than what we store in the artefact/display in the log viewer), since at the moment we display everything.

TBPL used to cap the number of failure lines shown in the brief log view to 100:
https://hg.mozilla.org/webtools/tbpl/file/4c71337583df/php/inc/LogParser.php#l40
...and the number of failures shown in the UI's failure summary to 20:
https://hg.mozilla.org/webtools/tbpl/file/4c71337583df/php/inc/AnnotatedSummaryGenerator.php#l25

Comment 1

[Ed Morley [:emorley]]

Going to steal this :-)

Comment 2

[Ed Morley [:emorley]]

We currently store the error lines in two places (see bug 1078450 comment 1 for example artefact):
* The artefact for the log viewer, that breaks the log into steps and lists the errors within that step.
* The error line/bug suggestion artefact

...so it's no wonder that the thousand lines of errors in comment 0 were taking up a fair bit of space in the job_artefacts table.

Comment 3

[Ed Morley [:emorley]]

Attachment: Limit the number of log error lines stored in the job artefact

Comment 4

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/45f291cb713fc4b8bdfe9069aa66498cf644bb43
Bug 1078415 - Log parser: Abstract out the step error list and length

https://github.com/mozilla/treeherder-service/commit/f9a0ddc85f88e4205773cbe1bb5de3de35582d17
Bug 1078415 - Log parser: Return early in parse_line() to simplify flow

If we're waiting for a step start line, we can just return early,
regardless of whether we found one or not, which means the later
check for |self.state == self.ST_STARTED| is now redundant. The previous
flow and comments were making the self.sub_parser.parse_line() call
look like it only occurred on the end step, when in fact it occurs only
when it's not the start _or_ end step.

https://github.com/mozilla/treeherder-service/commit/9ce4733946ef3b49de74d333ef513f9e427dc085
Bug 1078415 - Log parser: Only record the first 100 error lines per step

To avoid logs with excessive number of lines that match the error regex
from taking up too much space in the DB & also making the API response
and thus UI unwieldy, we cap the number of error lines at 100 per step
of the job. The 'errors_truncated' property can be used by the UI to
indicate that the error lines are only a subset of the total failures.