Closed Bug 1544460 Opened 7 years ago Closed 7 years ago

osx Mojave (10.14) - many reftests seem to fail

Categories

(Core :: Layout: Text and Fonts, defect, P3)

Product:
Core
Component:
Layout: Text and Fonts
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla69
Tracking Flags:
Tracking Status
firefox69 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

References

(Depends on 1 open bug)

Details

Attachments

(1 file)

Bug 1544460 - fix reftest annotations and enable for osx 10.14. r=jfkthame,kats
47 bytes, text/x-phabricator-request
Details | Review

there are many reftests failing while looking at tests running in automation on osx 10.14 (instead of the current 10.10 machines):
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher%40mozilla.com&fromchange=96a96f509d98f90ce3f8e35990df75153755b877&tochange=b9ef9517732230f87398e4b26d30ed5c2e34adce&searchStr=-reftest-e10s

If you look at the oranges, you will see many test-unexpected-fails.

I want to know if there is something we should be changing on the OS side (font installs, display resolution, etc?

Worse case scenario we just mark these all as fuzzy-if.

:jfkthame, could you take a look at a few failures and give some advice on what next steps would be here?

Flags: needinfo?(jfkthame)

It looks like an awful lot of failures here are related to font antialiasing (often of Ahem). In bug 1494715, we added an option to disable AA for Ahem in tests; is this perhaps not working on 10.14?

Mats, any chance you could check on this? (Though I don't know if you have a 10.14 system on hand -- I'm currently on 10.13 here.)

Flags: needinfo?(jfkthame) → needinfo?(mats)

Regarding the orthogonal-floats-1a.html/orthogonal-floats-1b.html tests: we should probably just remove the text inside the floats there since it doesn't really contribute to the test IMO. It seems we already tweaked them in the past:
https://searchfox.org/mozilla-central/source/layout/reftests/floats/orthogonal-floats-1a.html#48
(these tests have inline-size: 100px so removing the text shouldn't really affect the test)

orthogonal-floats-1c.html OTOH uses Ahem text to make a 100px content size, and yes it does look like we fail due to AA being enabled in the reftest-analyzer above.

I do have a MacBookAir with 10.14.3 on it, but when I load that test I don't see any difference with gfx.font_ahem_antialias_none enabled or not. (It's a 13-inch retina display and my eye-sight is rather poor so I can't see the individual pixels and the Screen Zoom feature blurs everything out). So the only way to test that is to run that test locally with/without that pref enabled, but I currently don't have a local build on this machine and it may take a while before I have time to set that up...

do you think pushing to try with some prefs toggled would answer these questions? I could do that if so

The first thing we should verify is that the pref is actually enabled in the test run above.
AFACT, it's enabled in testing/profiles/web-platform/user.js, is that file loaded for reftest runs? (the name seems to indicate it's only used for web-platform-tests?)
https://searchfox.org/mozilla-central/search?q=gfx.font_ahem_antialias_none
Is it possible to see the actual prefs we loaded in the logs above somehow?

Flags: needinfo?(jmaher)

so we only have the gfx.font_ahem_antialias_none for web-platform-tests; I can try this for reftests- do a before/after. As for the wpt-reftest failures, those might be fixed by the fact that we have hardcoded osx versions in many of the meta files.

Flags: needinfo?(jmaher)

Setting the pref is definitely an improvement IMO.
It fixes all of the Ahem AA issues, AFAICT.

The "variable-external-font-face" tests fails because they use a different family name for Ahem:
https://searchfox.org/mozilla-central/source/layout/reftests/w3c-css/submitted/variables/support/external-variable-font-face.css
and compare it to the -ref which use "Ahem" as the name, so it compares AA to non-AA.
Those should be fixed by annotating them with pref(gfx.font_ahem_antialias_none,false)
individually in the reftest.list manifest.

The remaining failures probably just needs a fuzzy() annotation (they have
max difference: 1 or 2).

(and as I said before, for the orthogonal-floats-1a.html/orthogonal-floats-1b.html
we should just remove the text inside the <div>s since it's not relevant for the test)

Joel, can you make a patch with those changes?

I'm not sure what to make of the three layout/reftests/font-face/color-* failures though.
Jonathan, do you have an idea?

Flags: needinfo?(mats)
Flags: needinfo?(jmaher)
Flags: needinfo?(jfkthame)

my understanding of the orthogonal tests is that the text between the div's is what we are testing, but the text outside the divs (in this case the <p>'s) is not related to the test.

:mats can you confirm that is what you want to see changed?

Flags: needinfo?(jmaher) → needinfo?(mats)

(In reply to Mats Palmgren (:mats) from comment #8)

I'm not sure what to make of the three layout/reftests/font-face/color-* failures though.

I think something about the rendering of COLR/CPAL-type color fonts has broken again. We had an issue like this previously in bug 1425812, but I've seen indications that some COLR fonts are again failing to render properly on more recent macOS versions; I thought we had a bug on file about this, but I can't seem to find it now.

My guess is that Core Text rendering behavior has changed underneath us in some way. Currently if I install the Twemoji Mozilla color font on my macOS 10.13 system, it just renders blanks instead of color glyphs. (On the other hand, https://www.harbortype.com/rocher-color/ renders fine.)

Given that I can't find an existing bug report, marking these as failing and filing a new bug on color font support seems like the best option; further investigation is needed.

Flags: needinfo?(jfkthame)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #9)

my understanding of the orthogonal tests is that the text between the div's is what we are testing, but the text outside the divs (in this case the <p>'s) is not related to the test.

:mats can you confirm that is what you want to see changed?

No. Specifically for orthogonal-floats-1a.html/orthogonal-floats-1b.html
(not the other ones), the text inside the <div>s is irrelevant for
the test because we specify both inline-size/block-size. These two
tests fail because a couple of AA pixels associated with the text
sticks out on the side. I'm expecting that by removing the text
they will pass. There's no need to touch the <p>'s or anything else
besides that.

Flags: needinfo?(mats)

(In reply to Jonathan Kew (:jfkthame) from comment #10)

(In reply to Mats Palmgren (:mats) from comment #8)

I'm not sure what to make of the three layout/reftests/font-face/color-* failures though.

I think something about the rendering of COLR/CPAL-type color fonts has broken again.

OK, I suggest we mark these as fail/random in the reftest.list manifest
for this platform then and file a bug on that (Graphics/Text).
Joel, can you take care of that please?

Flags: needinfo?(jmaher)

ok, I finally got a green try run:
https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=test-mac&selectedJob=241112427&revision=aa613a20edd11d5d230ef9ac74dfd17d15389ed1

there are about 340 changes to manifests (many are tests that now pass or are much less fuzzy):
https://hg.mozilla.org/try/rev/6ab3ef25822013047e61be4fcfc020d77eb1e6b1

Flags: needinfo?(jmaher)

remaining work:

  1. confirm I did variable-external-font-face properly in the patch as well as pref(gfx.font_ahem_antialias_none,false)
  2. edit orthogonal-floats-1a.html/orthogonal-floats-1b.html (should be independent of this bug)
  3. file a new bug to look at layout/reftests/font-face/color-*
  4. look for other areas in the edited files that might be concerning
Depends on: 1545419

here are all the changes I would like confirmation on before having a patch ready for review:

diff --git a/layout/reftests/bugs/reftest.list b/layout/reftests/bugs/reftest.list
-fuzzy-if(webrender&&winWidget,82-82,76-76) == 513153-2a.html 513153-2-ref.html
+fuzzy-if(webrender&&winWidget,82-82,76-76) (fuzzy-if(webrender&&OSX,29-29,112-112) == 513153-2a.html 513153-2-ref.html

diff --git a/layout/reftests/forms/input/file/reftest.list b/layout/reftests/forms/input/file/reftest.list
-fuzzy-if(Android,0-2,0-2) fails-if(webrender&&!cocoaWidget) == dynamic-max-width.html dynamic-max-width-ref.html # bug 1496542 for webrender.
+fuzzy-if(Android,0-2,0-2) fails-if(webrender&&!cocoaWidget) fuzzy-if(webrender&&cocoaWidget,34-34,134-134) == dynamic-max-width.html dynamic-max-width-ref.html # bug 1496542 for webrender.

diff --git a/layout/reftests/forms/progress/reftest.list b/layout/reftests/forms/progress/reftest.list
-fails-if(!cocoaWidget||OSX==1010) != progress-orient-vertical.html progress-orient-vertical-rtl.html # only OS X currently has direction-dependent rendering here
+fails != progress-orient-vertical.html progress-orient-vertical-rtl.html # only OS X currently has direction-dependent rendering here

diff --git a/layout/reftests/text-svgglyphs/reftest.list b/layout/reftests/text-svgglyphs/reftest.list
-pref(gfx.font_rendering.opentype_svg.enabled,true) fuzzy-if(webrender&&!gtkWidget,129-138,2188-2461) == svg-glyph-invalid.html svg-glyph-invalid-ref.html
+pref(gfx.font_rendering.opentype_svg.enabled,true) fuzzy-if(webrender&&!gtkWidget,129-138,2188-2461) fuzzy-if(OSX,134-134,2463-2463) == svg-glyph-invalid.html

diff --git a/layout/reftests/transform-3d/reftest.list b/layout/reftests/transform-3d/reftest.list
-fuzzy-if(gtkWidget,0-128,0-100) fuzzy-if(Android||OSX==1010||(gtkWidget&&layersGPUAccelerated),0-143,0-100) fuzzy-if(winWidget||OSX<1010,0-141,0-100) == preserves3d-nested.html preserves3d-nested-ref.html
+fuzzy-if(gtkWidget,0-128,0-100) fuzzy-if(Android||OSX==1010||(gtkWidget&&layersGPUAccelerated),0-143,0-100) fuzzy-if(winWidget||OSX,0-141,0-100) == preserves3d-nested.html preserves3d-nested-ref.html

diff --git a/layout/reftests/w3c-css/failures.list b/layout/reftests/w3c-css/failures.list
-fuzzy-if(Android,78,179) css-values/ch-unit-003.html
+fuzzy-if(OSX||Android,0-78,0-179) css-values/ch-unit-003.html

-fuzzy-if(OSX||winWidget,0-75,0-404) fuzzy-if(webrender&&!gtkWidget,92-108,300-404) css-writing-modes/direction-v??-00?.xht
-fuzzy-if(OSX||winWidget,0-135,0-904) css-writing-modes/float-contiguous-v??-01?.xht
-fuzzy-if(OSX||winWidget,0-109,0-300) css-writing-modes/float-shrink-to-fit-vlr-009.xht
-fuzzy-if(OSX||winWidget,0-109,0-300) css-writing-modes/float-shrink-to-fit-vrl-008.xht
-fuzzy-if(OSX||winWidget,0-108,0-300) css-writing-modes/float-v??-0??.xht
-fuzzy-if(OSX||winWidget,0-62,0-404) fuzzy-if(webrender&&winWidget,92-92,100-100) fails-if(webrender&&cocoaWidget) css-writing-modes/height-width-inline-non-replaced-v??-00?.xht
+fuzzy-if(!webrender&&OSX,255-255,200-200) fuzzy-if(winWidget,0-75,0-404) fuzzy-if(webrender&&winWidget,92-108,300-404) css-writing-modes/direction-v??-00?.xht
+fuzzy-if(OSX||winWidget,0-255,0-904) css-writing-modes/float-contiguous-v??-01?.xht
+fuzzy-if(!webrender&&OSX,255-255,100-100) fuzzy-if(winWidget,0-109,0-300) css-writing-modes/float-shrink-to-fit-vlr-009.xht
+fuzzy-if(!webrender&&OSX,255-255,100-100) fuzzy-if(winWidget,0-109,0-300) css-writing-modes/float-shrink-to-fit-vrl-008.xht
+fuzzy-if(!webrender&&OSX,255-255,49-150) fuzzy-if(winWidget,0-108,0-300) css-writing-modes/float-v??-0??.xht
+fuzzy-if(!webrender&&OSX,255-255,100-100) fuzzy-if(winWidget,0-62,0-404) fuzzy-if(webrender&&winWidget,92-92,100-100) css-writing-modes/height-width-inline-non-replaced-v??-00?.xht
-fuzzy-if(OSX||winWidget,0-135,0-1080) css-writing-modes/inline-block-alignment-slr-009.xht
-fuzzy-if(OSX||winWidget,0-112,0-960) css-writing-modes/inline-block-alignment-srl-008.xht
+fuzzy-if(!webrender&&OSX,255-255,432-432) fuzzy-if(winWidget,0-135,0-1080) css-writing-modes/inline-block-alignment-slr-009.xht
+fuzzy-if(!webrender&&OSX,255-255,432-432) fuzzy-if(winWidget,0-112,0-960) css-writing-modes/inline-block-alignment-srl-008.xht
-fuzzy-if(OSX||winWidget,0-75,0-404) fuzzy-if(webrender&&!gtkWidget,92-108,300-404) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-align-v??-0??.xht
-fuzzy-if(OSX||winWidget,0-215,0-780) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-baseline-???-00?.xht
-fuzzy-if(OSX,0-23,0-16) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-combine-upright-decorations-001.html
+fuzzy-if(!webrender&&OSX,255-255,200-200) fuzzy-if(winWidget,0-75,0-404) fuzzy-if(webrender&&winWidget,92-108,300-404) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-align-v??-0??.xht
+fuzzy-if(OSX||winWidget,0-255,0-780) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-baseline-???-00?.xht
+fuzzy-if(!webrender&&OSX,255-255,100-100) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-combine-upright-decorations-001.html
-fuzzy-if(OSX||winWidget,0-226,0-960) fails-if(webrender&&cocoaWidget) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-orientation-016.xht
-fuzzy-if(OSX||winWidget,0-223,0-720) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/vertical-alignment-.xht
-fuzzy-if(OSX||winWidget,0-158,0-624) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/writing-mode-vertical-??-00?.
+fuzzy-if(!webrender&&OSX,255-255,64-64) fuzzy-if(winWidget,0-226,0-960) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/text-orientation-016.xht
+fuzzy-if(winWidget,0-223,0-720) fuzzy-if(!webrender&&OSX,255-255,120-200) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/vertical-alignment-.xht
+fuzzy-if(winWidget,0-158,0-624) fuzzy-if(!webrender&&OSX,255-255,480-520) random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/writing-mode-vertical-??-00?.
-fuzzy-if(OSX||winWidget,0-112,0-960) css-writing-modes/inline-block-alignment-006.xht
+fuzzy-if(!webrender&&OSX,255-255,432-432) fuzzy-if(winWidget,0-112,0-960) css-writing-modes/inline-block-alignment-006.xht
-random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) css-writing-modes/outline-inline*-vrl-006.html
+random-if(/^Windows\x20NT\x206.1/.test(http.oscpu)) fuzzy-if(!webrender&&OSX,255-255,100-150) css-writing-modes/outline-inline*-v??-006.html

diff --git a/layout/reftests/w3c-css/submitted/variables/reftest.list b/layout/reftests/w3c-css/submitted/variables/reftest.list
-== variable-external-font-face-01.html variable-external-font-face-01-ref.html
+pref(gfx.font_ahem_antialias_none,false) fuzzy-if(OSX,48-49,622-622) == variable-external-font-face-01.html variable-external-font-face-01-ref.html
-== variable-font-face-01.html variable-font-face-01-ref.html
-== variable-font-face-02.html variable-font-face-02-ref.html
+fuzzy-if(OSX,48-49,622-622) == variable-font-face-01.html variable-font-face-01-ref.html
+fuzzy-if(OSX,48-49,622-622) == variable-font-face-02.html variable-font-face-02-ref.html

diff --git a/layout/tools/reftest/runreftest.py b/layout/tools/reftest/runreftest.py

  •    prefs['gfx.font_ahem_antialias_none'] = True

:jfkthame, could you comment on some of the changes here and recommend someone to review others you might not know of?

Flags: needinfo?(jfkthame)

the orthogonal-floats-1a.html/orthogonal-floats-1b.html tests seem to fail on all platforms if I remove the <div>'s:
https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=reftest&revision=e9d65f97847cd1b3725f0f6677da38af74d3e56a

and setting the gfx.font_ahem_antialias_none=true in case that was needed (yields many other failures- more passes):
https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=reftest&revision=3af44e2d81f390ccbf01eb9721cc0d9ded70e1be

here are the changes I made to get the tests (without editing them) passing on osx 10.14:
-fuzzy-if(OSX==1010,0-28,0-11) fuzzy-if(Android,0-16,0-2) == orthogonal-floats-1a.html orthogonal-floats-1-ref.html
-fuzzy-if(OSX==1010,0-28,0-11) fuzzy-if(winWidget,0-137,0-3) == orthogonal-floats-1b.html orthogonal-floats-1-ref.html
-fuzzy-if(OSX==1010,0-103,0-802) fuzzy-if(winWidget,0-135,0-700) == orthogonal-floats-1c.html orthogonal-floats-1-ref.html
-fuzzy-if(OSX==1010,0-103,0-802) fuzzy-if(winWidget,0-135,0-700) == orthogonal-floats-1d.html orthogonal-floats-1-ref.html
+fuzzy-if(OSX,0-28,0-11) fuzzy-if(Android,0-16,0-2) == orthogonal-floats-1a.html orthogonal-floats-1-ref.html
+fuzzy-if(OSX,0-28,0-11) fuzzy-if(winWidget,0-137,0-3) == orthogonal-floats-1b.html orthogonal-floats-1-ref.html
+fuzzy-if(winWidget,0-135,0-700) == orthogonal-floats-1c.html orthogonal-floats-1-ref.html
+fuzzy-if(winWidget,0-135,0-700) == orthogonal-floats-1d.html orthogonal-floats-1-ref.html

:mats, should we ignore the edits to the files? do you have other changes I should consider?

Flags: needinfo?(mats)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #16)

the orthogonal-floats-1a.html/orthogonal-floats-1b.html tests seem to fail on all platforms if I remove the <div>'s:
https://treeherder.mozilla.org/#/jobs?repo=try&searchStr=reftest&revision=e9d65f97847cd1b3725f0f6677da38af74d3e56a

We should remove the text inside those <div>'s, not remove the <div>'s themselves.

Flags: needinfo?(mats)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #15)

:jfkthame, could you comment on some of the changes here and recommend someone to review others you might not know of?

These look reasonable to me. (I'm sad that we have to add a bunch of annotations like this, but realistically I don't think it's a priority to dig deeper into these at the moment; they don't look like "real" functional bugs that we need to fix.)

Flags: needinfo?(jfkthame)

last question to resolve here-
do we want to set gfx.font_ahem_antialias_none=true for all reftests on all platforms? We talk about setting this here for osx for the entire reftest suite.

Flags: needinfo?(jfkthame)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #19)

last question to resolve here-
do we want to set gfx.font_ahem_antialias_none=true for all reftests on all platforms? We talk about setting this here for osx for the entire reftest suite.

I suspect it'd probably be a good idea; in general it's more likely to help than hurt.

Maybe file a separate bug about deploying such a change, and see what tryserver thinks of it? You could probably do that ahead of time, before the macOS upgrade lands.

Flags: needinfo?(jfkthame)

yeah, that is my plan, solve all the general stuff first, then have a smaller patch for OSX 10.14 specifics.

Depends on: 1545859
Priority: -- → P3

for reference, It will be at least 2 weeks before we are ready to move machines, so I will have a specific patch ready in mid may.

fix reftest annotations and enable for osx 10.14.

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/61539258029a fix reftest annotations and enable for osx 10.14. r=jfkthame,kats

https://hg.mozilla.org/mozilla-central/rev/61539258029a

Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox69: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Assignee: nobody → jmaher
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: