Reftest failures of the form "testfilename.html == referencefilename.html" should search for bugs with testfilename.html

RESOLVED FIXED

Status

Tree Management
Treeherder
--
major
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: philor, Assigned: jgraham)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
Finally realized why webgl reftest failures are now all getting "too many bug suggestions": we switched reftest logging to list both the test filename and the reference filename, like "REFTEST PROCESS-CRASH | http://10.0.2.2:8854/tests/dom/canvas/test/reftest/webgl-resize-test.html == http://10.0.2.2:8854/tests/dom/canvas/test/reftest/wrapper.html?green.png | application crashed [@ jemalloc_crash]"", and so when treeherder looks for the test filename, it takes the thing after the rightmost / between the first and second |, which is wrapper.html?green.png, which is the reference file for tens or hundreds of different webgl reftests.

The "everything should be fixed upstream" view of that would be that the ouput should instead be "REFTEST BADTHING | tiresomepath/testfilename | == referencefilename bad thing that happened", that the test is testfilename and the == referencefilename bit is the equivalent of an individual is(this, that) in a mochitest.

Or, treeherder could just hack up the hacky filename finder to know that for reftests it's not the rightmost, it's the rightmost to the left of the first ==/!=.

Either way, this is a huge regression, and the reason why I've been filing five or six or ten duplicates of the same webgl crash lately.
Created attachment 8777969 [details] [review]
[treeherder] mozilla:error_summary_reftest > mozilla:master
(Assignee)

Updated

2 years ago
Attachment #8777969 - Flags: review?(emorley)

Comment 2

2 years ago
Comment on attachment 8777969 [details] [review]
[treeherder] mozilla:error_summary_reftest > mozilla:master

This is a deviance from the (albeity crufty) log format we've used for years.
I'm not really sure we should be working around this in Treeherder.

If people want to propose a new error message format for unstructured logs, they are welcome to, but let's have proper mailing list discussion, choice of new format and support added to all places where error lines are parsed first.
Attachment #8777969 - Flags: review?(emorley)
(Assignee)

Comment 3

2 years ago
Comment on attachment 8777969 [details] [review]
[treeherder] mozilla:error_summary_reftest > mozilla:master

Per the conversation on irc it sounds like you might be OK with this after all.
Attachment #8777969 - Flags: review?(emorley)

Updated

2 years ago
Attachment #8777969 - Flags: review?(emorley) → review+

Updated

2 years ago
Assignee: nobody → james

Comment 4

2 years ago
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/be1ca5872bb56ba78fb92b7a03fb3bfccd6a5ca7
Bug 1292338 - Remove reftest references from error summary search terms. (#1761)

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.