Closed Bug 1401997 Opened 3 years ago Closed 3 years ago

Report the reftest difference for fails-if tests that do fail

Categories

(Testing :: Reftest, enhancement)

57 Branch
enhancement
Not set

Tracking

(firefox58 fixed)

RESOLVED FIXED
mozilla57
Tracking Status
firefox58 --- fixed

People

(Reporter: kats, Assigned: kats)

Details

Attachments

(1 file)

Right now, when reftests are run, we report the pixel differences for tests marked fuzzy, and for tests that result in UNEXPECTED-FAIL. I think we should just always report it.

In particular, it's useful to see if tests marked as fails[-if] are actually very close to passing (e.g. they have a 1-pixel difference). In such cases the harness will treat the test as TEST-KNOWN-FAIL which is fine but if the difference is small the test could just as well be annotated as fuzzy-if and be made passing.
Comment on attachment 8910786 [details]
Bug 1401997 - Always report pixel difference counts for reftests.

https://reviewboard.mozilla.org/r/182246/#review187586

this seems very useful :)
Attachment #8910786 - Flags: review?(jmaher) → review+
Pushed by kgupta@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2b87ee06ac04
Always report pixel difference counts for reftests. r=jmaher
https://hg.mozilla.org/mozilla-central/rev/2b87ee06ac04
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
You need to log in before you can comment on or make changes to this bug.