974243 - [Tracking] (reftest) tests in layout/reftests/reftest-sanity failed

Status: RESOLVED WONTFIX

(Testing :: Reftest, defect)

Description

[Vincent Chen [:vichen]]

Attachment: reftest-sanity.html

Comment 1

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

A few comments about the bugs being filed:

 * reftest logs (containing the image output on failures) are substantially more useful, since it allows others to see the failure in reftest-analyzer.xhtml

 * is it really necessary to have one bug report per directory?  It seems more useful to group the bugs based on characteristics of the failures, especially if (as I would expect with a large number of failures) there are some significant groups of failures with the same root cause.

 * it's a bit confusing to call directories "modules"; I'd much prefer "tests in layout/reftests/bugs" over "tests of module bugs"

Comment 2

[Vincent Chen [:vichen]]

Attachment: reftest-sanity.log

Comment 3

[Vincent Chen [:vichen]]

(In reply to David Baron [:dbaron] (needinfo? me) (UTC-8) from comment #1)
> A few comments about the bugs being filed:
> 
>  * reftest logs (containing the image output on failures) are substantially
> more useful, since it allows others to see the failure in
> reftest-analyzer.xhtml

log added

> 
>  * is it really necessary to have one bug report per directory?  It seems
> more useful to group the bugs based on characteristics of the failures,
> especially if (as I would expect with a large number of failures) there are
> some significant groups of failures with the same root cause.

1. Bug 973835 is meta bug. Seems to have many root cause of these reftest fail, use different bugs by test directory to keep tracking. 
2. Will provide a patch by skipping current fail and enable B2G reftest (Bug 922680). To skip a test need an associated bug number with it.
3. Will group bugs based on characteristics again. 
4. Currently, font-inflation fail (Bug 972697) is due to not applying font-inflation during reftest, if roll-back https://bugzilla.mozilla.org/show_bug.cgi?id=922680#c32 font-inflation will be applied and all test in font-inflation will pass and (Bug 974214) reftests/scrolling/uncovering-1.html will pass, too. But it introduces another reftest in reftests/scrolling fail. We're still examining root cause.



> 
>  * it's a bit confusing to call directories "modules"; I'd much prefer
> "tests in layout/reftests/bugs" over "tests of module bugs"

agree and all title updated.

Comment 4

[(not currently active) Ted Mielczarek]

The test failure in this log is:
REFTEST TEST-UNEXPECTED-FAIL | http://10.0.1.29:8888/tests/layout/reftests/reftest-sanity/656041-1.html | image comparison (==), max difference: 4, number of differing pixels: 2408

The actual difference seems to be that the scrollbar on the right hand side is slightly different colors in the test and reference.