1367745 - [mozharness] Log 'process_output' messages at the proper mozharness level with StructuredOutputParser

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, enhancement)

Description

[Andrew Halberstadt [:ahal]]

In the StructuredOutputParser, we pass 'data' portion of 'process_output' messages to the BaseOutputParser so we can run the regex error lists against them.

But we don't save what the computed log level was, which means they'll always be logged at INFO (even if an error was detected). Because of this, treeherder won't include these lines in its error summary.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1367745 - Use same error lists in wpt as in mochitest,

Review commit: https://reviewboard.mozilla.org/r/142894/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/142894/

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1367745 - Make sure 'process_output' messages are logged at the proper mozharness level,

In addition to saving the log level of 'process_output' messages, this will also start passing 'log'
messages through the error lists. This means mozharness will start using 'log' errors when determining
the tbpl_status and worst_log_level.

Review commit: https://reviewboard.mozilla.org/r/142896/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/142896/

Comment 3

[Andrew Halberstadt [:ahal]]

Because this touches logging and can therefore break random tasks, here's a fairly comprehensive try run:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]

Comment 4

[James Graham [:jgraham]]

Comment on attachment 8871419 [details]
Bug 1367745 - Use same error lists in wpt as in mochitest,

https://reviewboard.mozilla.org/r/142894/#review146928

Comment 5

[James Graham [:jgraham]]

Comment on attachment 8871420 [details]
Bug 1367745 - Make sure 'process_output' messages are logged at the proper mozharness level,

https://reviewboard.mozilla.org/r/142896/#review146930

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ad1fa17ce50e
Use same error lists in wpt as in mochitest, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/f19ff8201437
Make sure 'process_output' messages are logged at the proper mozharness level, r=jgraham

Comment 7

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b56fa748e7ec
Backed out changeset f19ff8201437 for causing Windows debug reftests to exceed the maximum log size.

Comment 8

[Andrew Halberstadt [:ahal]]

The super().parse_single_line function was returning INFO by default, which is technically "worse" than DEBUG. So things logged at the debug level were being transformed to INFO.

The fix is to get the super().parse_single_line to only return a level if there is a match in the error_list, and otherwise return None. That way we can only overwrite the log level if there was a match of the error_list.

Comment 9

[Andrew Halberstadt [:ahal]]

Fixed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4545794fceb3ba3469b1c0d453de697ae92fbb0d

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ad1fa17ce50e

Comment 11

[Kartikaya Gupta (email:[e-mail])]

Reopening since one of the two patches was backed out and needs relanding.

Comment 12

[Andrew Halberstadt [:ahal]]

Comment on attachment 8871420 [details]
Bug 1367745 - Make sure 'process_output' messages are logged at the proper mozharness level,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/142896/diff/1-2/

Comment 13

[Andrew Halberstadt [:ahal]]

Comment on attachment 8871420 [details]
Bug 1367745 - Make sure 'process_output' messages are logged at the proper mozharness level,

Asking for re-review as the fix was just complex enough to make me uncomfortable.

Comment 14

[Andrew Halberstadt [:ahal]]

In addition to the previous try run, here's another general one just to be extra safe:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=719616bbce7ab885491f53379b708ce402b68b41

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/454c9ae2e8ba
Make sure 'process_output' messages are logged at the proper mozharness level, r=jgraham

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/454c9ae2e8ba