Open Bug 654218 Opened 9 years ago Updated 7 years ago

reftest harness prints only one images for tests which pass unexpectedly


(Testing :: Reftest, defect)

Not set


(Not tracked)


(Reporter: ehsan, Unassigned)


STR: run reftests in layout/reftests/bugs/reftest.list and make sure the browser window doesn't have the focus.  You will get this in the logs:

REFTEST TEST-START | file:///Users/ehsanakhgari/moz/mozilla-central/layout/reftests/bugs/503531-1.html
REFTEST TEST-START | file:///Users/ehsanakhgari/moz/mozilla-central/layout/reftests/bugs/503531-1-ref.html
EFTEST TEST-UNEXPECTED-PASS | file:///Users/ehsanakhgari/moz/mozilla-central/layout/reftests/bugs/503531-1.html | image comparison (==)
REFTEST   IMAGE: data:image/png;base64,...
REFTEST INFO | Loading a blank page

This confuses reftest-analyzer.xhtml...
Since an unexpected pass on a "==" comparison means the two images are identical, I think it's sensible enough that it only prints it out once.  What confusion does it cause reftest-analyzer.xhtml?
I thought reftest-analyzer.xhtml handled this fine.  In any case, the behavior of the harness is as-expected.
I guess I've reported this in another way in the past.  reftest-analyzer treats this type of situation by just showing one image, which has confused me on multiple occasions.  I'd rather see two identical images, with maybe a piece of text explicitly saying that they're identical.  But if you both think this is a better UI, then I think I'll probably just change my local copy...
I think it would make sense for reftest-analyzer to have something at the top that says "One identical image" or something like that (instead of the selector for flipping between them).  That would be a useful reminder.
You need to log in before you can comment on or make changes to this bug.