Open Bug 1463844 Opened 6 years ago Updated 2 years ago

Lots of false positive test failures for reftests tests on Mac OS due to image differences


(Testing :: web-platform-tests, defect)

Version 3


(Not tracked)


(Reporter: whimboo, Unassigned)


There are lots of failures on Mac OS for web-platform-tests reftests but not on other platforms. For now all of those tests have been marked as expected fail. Failures look like:

> 1:40.10 TEST_END: FAIL, expected PASS - Testing http://web-platform.test:8000/webvtt/rendering/cues-with-video/processing-model/selectors/default_styles/italic_object_default_font-style.html == http://web-platform.test:8000/webvtt/rendering/cues-with-video/processing-model/selectors/default_styles/italic_object_default_font-style-ref.html

Examples can be found under:

As I just noticed those failures seem to depend on how you run those tests. When I worked on bug 1463781 the tests were run on a secondary display with normal resolution. All the tests were passing with the attached patch.

A couple of minutes ago I tried to run the same tests on my Macbook's internal display (while unplugged from the secondary screen) and lots of them unexpectedly are passing.

So maybe the problem we see with the failures depends on running the tests on a Retina display or not? Without the Retina display we seem to fail a lot more.
As it looks like the underlying problem here is related to the following flag which is used when taking screenshots with reftests:


Removing that causes tests to pass for me. Also see the normal reftests in regards of this flag:

I will have another look into that tomorrow.
My last comment was red herring. The reason why it is passing all the time is that the taken screenshots are blank and as such are equal. I will take some example screenshots from the webvtt subfolder which contains a couple of tests which always fail even when running just one test.
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.