1517264 - Add reftest-analyzer support for wrench reftests

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, enhancement, P3)

Description

[Kartikaya Gupta (email:[e-mail])]

Right now if you have reftest failures in wrench on treeherder (e.g. [1]) the reftest analyzer is not shown. This requires a patch to treeherder to recognize the wrench jobs as a "reftest" type job. But also, once that is done, the reftest analyzer doesn't successfully parse the log. So that requires another patch.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=c4eec6516d64ae6320f0e3169996039157656dd2&selectedJob=219176335

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1517264 - Add a mechanism to regression-test the reftest parsing code. r?dbaron

This code has gotten tweaked a number of times over the years to support
new log formats, but there's always the risk that a tweak will break old
log formats. So this patch adds a small test function that can be run by
anybody making changes to ensure old logs still parse fine. A few
current log samples are tested thusly.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1517264 - Add support for wrench reftest parsing in the reftest-analyzer. r?dbaron

The wrench reftest suite runs reftests on the standalone webrender
project, and emits output that is compatible with the reftest analyzer.
However, the output takes a slightly different path through taskcluster
(the job runs via generic-worker rather than mozharness) and so doesn't
have the "(INFO|ERROR) -" prefix that the analyzer expects. This patch
loosens that restriction so that these logs can be parsed properly.

Depends on D15576

Comment 3

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozilla/treeherder/pull/4412 is the treeherder PR

Comment 4

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/ebe9db4b7cb94b29793fd53348c543f26ac776fa
Bug 1517264 - Treat wrench jobs as reftests. (#4412)

Wrench jobs are a job type for the standalone WebRender test suite, and
run reftests as part of the job. Having reftest-ish things like the
reftest analyzer links for these jobs is desirable.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2c549bcb88d4
Add a mechanism to regression-test the reftest parsing code. r=dbaron
https://hg.mozilla.org/integration/autoland/rev/fd129c882b77
Add support for wrench reftest parsing in the reftest-analyzer. r=dbaron

Comment 6

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/2c549bcb88d4
https://hg.mozilla.org/mozilla-central/rev/fd129c882b77