1276486 - Add 'extra_highlights' field to the job object

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: API, defect)

Description

[Andrew Halberstadt [:ahal]]

It would be awesome if we could define a list of extra regexes to use to determine which log lines to highlight directly in the taskcluster configs.

Currently, jobs need to use ugly log formatters so that treeherder will recognize which lines to highlight. For example:
https://dxr.mozilla.org/mozilla-central/source/tools/lint/eslint-formatter.js

The default eslint formatter is much more readable, unfortunately we can't use it. This ability would also allow us to move off the mozlog tbplformatter without requiring treeherder<->mozlog integration (at least for taskcluster jobs).

Comment 1

[Ed Morley [:emorley]]

I don't think we want to do this. 

This would need a bit of a rewrite of the log parser (and log viewer), plus is meaning we'd be storing error handling logic in the Treeherder DB rather than only in code -- which just seems pretty wrong at first glance.

To me, preferable solutions would be:
1) Submitters do their own error summarisation (in the future we see Treeherder not parsing logs at all)
2) Submitting the logs as structured logs (we really should be reducing the number of jobs that use unstructured logs, not increasing them).
3) Using mozharness (or is it mozlog) style prefixed logs (ie with the error level prefix), such that Treeherder's log parser doesn't need a hardcoded list of regex, but simply matches against the ERROR level lines
4) (As a last resort) Adding to Treeherder's existing regex as a one-off

Open to further reasoning though! ;-)

Comment 2

[Ed Morley [:emorley]]

s/;-)/:-)/

Comment 3

[Andrew Halberstadt [:ahal]]

(In reply to Ed Morley [:emorley] from comment #1)
> This would need a bit of a rewrite of the log parser (and log viewer), plus
> is meaning we'd be storing error handling logic in the Treeherder DB rather
> than only in code -- which just seems pretty wrong at first glance.

To clarify, my hope was to get taskcluster to pass this data in via API call, e.g here:
https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/tests/eslint-gecko.yml#36

I didn't think that would need to get stored in a DB anywhere, but I have 0 knowledge of treeherder's architecture :). I may have also horribly mis-worded the title. I was hoping this would be an easy fix, but if it's not then I guess we can close this..


> To me, preferable solutions would be:
> 1) Submitters do their own error summarisation (in the future we see
> Treeherder not parsing logs at all)

I'd love to be able to do this, but I didn't think it was possible. In fact, this is the problem I was trying to solve with this bug! Is there a way to do this now?


> 2) Submitting the logs as structured logs (we really should be reducing the
> number of jobs that use unstructured logs, not increasing them).

For the record, the jobs I'm thinking about are lint jobs which don't really make sense under the mozlog model (there is no concepts of "suite" or "test"). We could probably shoe-horn it in, but it would be kind of awkward. The lint jobs will be able to produce json-lines, but it will be a different structure than mozlog raw.

Comment 4

[William Lachance (:wlach)]

Could you describe in some more detail the user-facing behaviour you want to enable? I don't know what you mean by "highlight". Do you mean specify error lines that will get shown in the treeherder jobs panel, log viewer, or both?

It is indeed possible to do your own log parsing at the moment and submit your own error summary, but the procedure is undocumented and I'm not sure if I recommend it.

Comment 5

[Andrew Halberstadt [:ahal]]

Both I suppose, I sort of figured they used the same set of regexes. But yes, the blue "Failure summary" panel and the lines in the log viewer.

Comment 6

[Ed Morley [:emorley]]

I don't think we should do this for the reasons in comment 1.