1633504 - reftest-analyzer disagrees with TreeHerder logs on differing pixel count for fuzzy failures

Status: RESOLVED FIXED

(Testing :: Reftest, defect, P2)

Description

[Daniel Holbert [:dholbert]]

STR:
(1) View this reftest log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=299100920&repo=mozilla-beta&lineNumber=4432 and note how many pixels are reported as differing.
(2) Click "open analyzer" (or visit direct link)
(3) Click the "outline.html == [etc]` text for the reftest failure, to show the mismatching image view.
(4) Scroll down.

ACTUAL RESULTS:
The reftest log says: max difference: 7, number of differing pixels: 214
...whereas reftest-analyzer says: Maximum difference per channel 7, 140 pixels differ

EXPECTED RESULTS:
They should agree on how many pixels differ.

(It looks like this numerical disagreement is unexpected, per bug 1614904 comment 2 - 3)

Comment 1

[Daniel Holbert [:dholbert]]

I saw this exact issue for 4 logs that were all about different occurrences of this same sort of failure in this particular layout/reftests/svg/outline.html reftest:

https://treeherder.mozilla.org/logviewer.html#?job_id=299100920&repo=mozilla-beta [this is the log from comment 0]
https://treeherder.mozilla.org/logviewer.html#?job_id=298861634&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=298559973&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=298417320&repo=mozilla-central

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Interesting. When I load the reftest analyzer I get 7,214 as expected, using FF 75.0 build 20200331175109 on macOS with webrender enabled. On latest Nightly with WR enabled, same machine, I get 7,139 as the difference.

Comment 3

[James Graham [:jgraham]]

The only difference I'm aware of is that the reftest-analyzer ignores the alpha channel, assuming there aren't any transparent pixels. But changing that locally didn't make any difference in one of the problematic cases.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

Do you have cycles to take this bug and investigate further? If not maybe we can disable this feature for now. I'm not sure it's worth displaying those numbers if they can't be relied on.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

I'll probably disable the feature later this week if I don't hear back, assuming you won't have time in the near future to investigate.

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1633504 - Disable showing pixel difference numbers in the reftest analyzer. r?dholbert

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/594d2cecc8cf
Disable showing pixel difference numbers in the reftest analyzer. r=dholbert

Comment 8

[Emilio Cobos Álvarez (:emilio)]

Kats, one issue is that WPT reftest don't show the pixel count in the log, so this is the only way to get information from them. And the fuzzy match probably matches the WPT version. So this count is useful if you're looking at WPT reftests instead of regular reftests.

Comment 9

[James Graham [:jgraham]]

wpt does show it in the logs e.g. [1], although perhaps it's harder to find. And on the backend the wpt reftests are using the same code as the reftest harness reftests, so there shouldn't be a difference there. I didn't yet get a chance to investigate whether the problem is that the algorithm in the reftest analyzer is somehow different from the one that the harnesses use or if the problem is that we're outputting images that aren't exactly matching the internal canvas in some cases (or something else). The algorithm for comparison seems simple enough, and the numbers in the reftest analyzer are right in the cases that I've used so it's possible that there's something more subtle going on and this is only a symptom.

[1] https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=301478409&repo=mozilla-central&lineNumber=19204-19205

Comment 10

[Emilio Cobos Álvarez (:emilio)]

Ah, ok then.

Comment 11

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/594d2cecc8cf