Lots of false positive test failures for reftests tests on Mac OS due to image differences

NEW
Unassigned

Status

defect
Last year
Last year

People

(Reporter: whimboo, Unassigned)

Tracking

Version 3
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

There are lots of failures on Mac OS for web-platform-tests reftests but not on other platforms. For now all of those tests have been marked as expected fail. Failures look like:

> 1:40.10 TEST_END: FAIL, expected PASS - Testing http://web-platform.test:8000/webvtt/rendering/cues-with-video/processing-model/selectors/default_styles/italic_object_default_font-style.html == http://web-platform.test:8000/webvtt/rendering/cues-with-video/processing-model/selectors/default_styles/italic_object_default_font-style-ref.html

Examples can be found under:

https://dxr.mozilla.org/mozilla-central/source/testing/web-platform/tests/webvtt/rendering

As I just noticed those failures seem to depend on how you run those tests. When I worked on bug 1463781 the tests were run on a secondary display with normal resolution. All the tests were passing with the attached patch.

A couple of minutes ago I tried to run the same tests on my Macbook's internal display (while unplugged from the secondary screen) and lots of them unexpectedly are passing.

So maybe the problem we see with the failures depends on running the tests on a Retina display or not? Without the Retina display we seem to fail a lot more.
As it looks like the underlying problem here is related to the following flag which is used when taking screenshots with reftests:

> ctxInterface.DRAWWINDOW_USE_WIDGET_LAYERS

Removing that causes tests to pass for me. Also see the normal reftests in regards of this flag:

https://dxr.mozilla.org/mozilla-central/rev/9055d9d89a4bca5cf48dda789299559aefca4e54/layout/tools/reftest/reftest.jsm#806-814

I will have another look into that tomorrow.
My last comment was red herring. The reason why it is passing all the time is that the taken screenshots are blank and as such are equal. I will take some example screenshots from the webvtt subfolder which contains a couple of tests which always fail even when running just one test.
You need to log in before you can comment on or make changes to this bug.