654218 - reftest harness prints only one images for tests which pass unexpectedly

Status: NEW

(Testing :: Reftest, defect)

Description

[(no longer active)]

STR: run reftests in layout/reftests/bugs/reftest.list and make sure the browser window doesn't have the focus.  You will get this in the logs:

REFTEST TEST-START | file:///Users/ehsanakhgari/moz/mozilla-central/layout/reftests/bugs/503531-1.html
REFTEST TEST-START | file:///Users/ehsanakhgari/moz/mozilla-central/layout/reftests/bugs/503531-1-ref.html
EFTEST TEST-UNEXPECTED-PASS | file:///Users/ehsanakhgari/moz/mozilla-central/layout/reftests/bugs/503531-1.html | image comparison (==)
REFTEST   IMAGE: data:image/png;base64,...
REFTEST INFO | Loading a blank page

This confuses reftest-analyzer.xhtml...

Comment 1

[Cameron McCormack (:heycam)]

Since an unexpected pass on a "==" comparison means the two images are identical, I think it's sensible enough that it only prints it out once.  What confusion does it cause reftest-analyzer.xhtml?

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I thought reftest-analyzer.xhtml handled this fine.  In any case, the behavior of the harness is as-expected.

Comment 3

[(no longer active)]

I guess I've reported this in another way in the past.  reftest-analyzer treats this type of situation by just showing one image, which has confused me on multiple occasions.  I'd rather see two identical images, with maybe a piece of text explicitly saying that they're identical.  But if you both think this is a better UI, then I think I'll probably just change my local copy...

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I think it would make sense for reftest-analyzer to have something at the top that says "One identical image" or something like that (instead of the selector for flipping between them).  That would be a useful reminder.