Finally realized why webgl reftest failures are now all getting "too many bug suggestions": we switched reftest logging to list both the test filename and the reference filename, like "REFTEST PROCESS-CRASH | http://10.0.2.2:8854/tests/dom/canvas/test/reftest/webgl-resize-test.html == http://10.0.2.2:8854/tests/dom/canvas/test/reftest/wrapper.html?green.png | application crashed [@ jemalloc_crash]"", and so when treeherder looks for the test filename, it takes the thing after the rightmost / between the first and second |, which is wrapper.html?green.png, which is the reference file for tens or hundreds of different webgl reftests. The "everything should be fixed upstream" view of that would be that the ouput should instead be "REFTEST BADTHING | tiresomepath/testfilename | == referencefilename bad thing that happened", that the test is testfilename and the == referencefilename bit is the equivalent of an individual is(this, that) in a mochitest. Or, treeherder could just hack up the hacky filename finder to know that for reftests it's not the rightmost, it's the rightmost to the left of the first ==/!=. Either way, this is a huge regression, and the reason why I've been filing five or six or ten duplicates of the same webgl crash lately.
Created attachment 8777969 [details] [review] [treeherder] mozilla:error_summary_reftest > mozilla:master
Comment on attachment 8777969 [details] [review] [treeherder] mozilla:error_summary_reftest > mozilla:master This is a deviance from the (albeity crufty) log format we've used for years. I'm not really sure we should be working around this in Treeherder. If people want to propose a new error message format for unstructured logs, they are welcome to, but let's have proper mailing list discussion, choice of new format and support added to all places where error lines are parsed first.
Comment on attachment 8777969 [details] [review] [treeherder] mozilla:error_summary_reftest > mozilla:master Per the conversation on irc it sounds like you might be OK with this after all.
Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/be1ca5872bb56ba78fb92b7a03fb3bfccd6a5ca7 Bug 1292338 - Remove reftest references from error summary search terms. (#1761)
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.