1815965 - reduce output from unstructured logs in treeherder critical/error log parsing

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement, P1)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

typically in treeherder we parse and treat all errors and critical log messages as failures. For each failure we store lines and do a lot of database queries for matching and finding appropriate bugs.

It is easy for me to find wpt jobs that have:
Test harness output was not a valid structured log message

in many cases, this is either a python stack, or a bunch of unicode undecipherable bytes, such as:

127.0.0.1 - - [07/Feb/2023 20:29:23] code 400, message Bad request syntax ('\x16\x03\x01\x02\x00\x01\x00\x01ü\x03\x03ݳ\x91´,¤ø«8£g\x8ct\x86Z\x19Øú®\x90\x1e\x17\x94á\x04é\x0c\x89\x9fl\x90ó 7ê{/[ÿ÷¢Ú}äpäjÄݵ¯]í\x0b"¶\x97\x9b\x84\x90\x19¸$´§\x00\x1e\x13\x01\x13\x03\x13\x02À+À/̨̩À,À0À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x01\x00\x01\x95\x00\x00\x00\x16\x00\x14\x00\x00\x11web-platform.test\x00\x17\x00\x00ÿ\x01\x00\x01\x00\x00')
127.0.0.1 - - [07/Feb/2023 20:29:23] "

”áé
.
:

in the raw log all of these statements have CRITICAL message tags and they should have info tags, we can keep track of all of these failures based on Test harness output was not a valid structured log message This maps to bug 1273470.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1815965 - print out unstructured lines from wpt failures as INFO not CRITICAL. r=jgraham!

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/96ce3ab7ac4d
print out unstructured lines from wpt failures as INFO not CRITICAL. r=aryx

Comment 3

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/96ce3ab7ac4d

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

in looking at dozens of annotations from the sheriffs, they are not annotating on the single failure Test harness output was not a valid structured log message: remaining. In fact there are 4 instances annotated in the last 30 days, this remaining failure is noise, and we should treat it as a warning.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

see bug 1273470, bug 1440161, and bug 1688518

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1815965 - do not log unstructured output message as critical, only as info. r=aryx!

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/67b12d213d29
do not log unstructured output message as critical, only as info. r=aryx

Comment 8

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/67b12d213d29