885709 - Failure in layout/reftests/reftest-sanity/test-displayport-2.html is not analyzed by reftest analyzer

Status: RESOLVED INCOMPLETE

(Testing :: Reftest, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

See https://tbpl.mozilla.org/?tree=Try&rev=3056b44876ab - the reftest analyzer can't analyze the reftest failure there. The log looks something like this:

19:27:50     INFO -  REFTEST TEST-START | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/reftest-sanity/test-displayport-2.html
19:27:50     INFO -  REFTEST TEST-LOAD | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/reftest-sanity/test-displayport-2.html | 70 / 119 (58%)
19:27:51     INFO -  REFTEST   IMAGE: data:image/png;base64,iVBORw0KGgoAA...
19:27:51     INFO -  REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/reftest-sanity/test-displayport-2.html | image comparison (!=)
19:27:51     INFO -  REFTEST INFO | Saved log: START file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/reftest-sanity/test-displayport-2.html
19:27:51     INFO -  REFTEST INFO | Saved log: [CONTENT] Setting viewport to <w=100, h=100>

And I think the reftest analyzer can't deal with the REFTEST IMAGE line because it comes before the UNEXPECTED-FAIL line. I don't know if this is a bug in the analyzer or the reftest log file generator.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

If I paste the log into the reftest analyzer page manually with the image appearing after the unexpected-fail line then it works.

Comment 2

[Phil Ringnalda (:philor)]

Doesn't the reftest analyzer intentionally not try to deal with != tests, where there's only a single image in the log because the two were supposed to look different but failed to look different, so there's no difference between them which it can highlight?

Comment 3

[Kartikaya Gupta (email:[e-mail])]

I don't think so; just the other day I opened a reftest analyzer using the TBPL link and it showed me failures with just the one image.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

The format of reftest output has changed over the years. Now there's a test suite built into the reftest analyzer to prevent regressions, so we shouldn't have this problem again (or if we do, we can fix it once and for all).