1292338 - Reftest failures of the form "testfilename.html == referencefilename.html" should search for bugs with testfilename.html

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[Phil Ringnalda (:philor)]

Finally realized why webgl reftest failures are now all getting "too many bug suggestions": we switched reftest logging to list both the test filename and the reference filename, like "REFTEST PROCESS-CRASH | http://10.0.2.2:8854/tests/dom/canvas/test/reftest/webgl-resize-test.html == http://10.0.2.2:8854/tests/dom/canvas/test/reftest/wrapper.html?green.png | application crashed [@ jemalloc_crash]"", and so when treeherder looks for the test filename, it takes the thing after the rightmost / between the first and second |, which is wrapper.html?green.png, which is the reference file for tens or hundreds of different webgl reftests.

The "everything should be fixed upstream" view of that would be that the ouput should instead be "REFTEST BADTHING | tiresomepath/testfilename | == referencefilename bad thing that happened", that the test is testfilename and the == referencefilename bit is the equivalent of an individual is(this, that) in a mochitest.

Or, treeherder could just hack up the hacky filename finder to know that for reftests it's not the rightmost, it's the rightmost to the left of the first ==/!=.

Either way, this is a huge regression, and the reason why I've been filing five or six or ten duplicates of the same webgl crash lately.

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:error_summary_reftest > mozilla:master

Comment 2

[Ed Morley [:emorley]]

Comment on attachment 8777969 [details] [review]
[treeherder] mozilla:error_summary_reftest > mozilla:master

This is a deviance from the (albeity crufty) log format we've used for years.
I'm not really sure we should be working around this in Treeherder.

If people want to propose a new error message format for unstructured logs, they are welcome to, but let's have proper mailing list discussion, choice of new format and support added to all places where error lines are parsed first.

Comment 3

[James Graham [:jgraham]]

Comment on attachment 8777969 [details] [review]
[treeherder] mozilla:error_summary_reftest > mozilla:master

Per the conversation on irc it sounds like you might be OK with this after all.

Comment 4

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/be1ca5872bb56ba78fb92b7a03fb3bfccd6a5ca7
Bug 1292338 - Remove reftest references from error summary search terms. (#1761)