1481723 - Enable CSS tests on Windows/Mac

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[James Graham [:jgraham]]

On mac these were originally disabled for capacity reasons, and on Windows for some blank screenshots. However both of these issues are now solved.


A recent try run [1] shows two problems:

* On mac we sometimes get bogus results where after a fail tests are recorded as having pixel differences even though the reftest analyzer shows that they're identical

* On Windows we seem to have a lot of differences due to antialiasing.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a57bb4a7319cde5d8c3c5387ac45f655158b11e0&selectedJob=188953746

Comment 1

[Daniel Holbert [:dholbert]]

(In reply to James Graham [:jgraham] from comment #0)
> On mac these were originally disabled for capacity reasons,

bug 1425698 was an issue for wpt-on-mac, too, right?

> [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a57bb4a7319cde5d8c3c5387ac45f655158b11e0&selectedJob=188953746

(I was mildly curious to take a look at this Try run, but sadly, it looks like its log files have been purged due to our standard Try expiration processes.)

Comment 2

[James Graham [:jgraham]]

Yes, missing Ahem on Mac is an issue, but arguably not a blocker since it "just" means that tests using Ahem as a non-webfont give an incorrect result on mac (of course we should fix that but it's possible to work around by using webfonts or similar).

A more recent try run where I added all the prefs that are set in the reftest harness but not in wpt, in case that fixed any issues (it didn't):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b020d7b8b1c8ce716140f39eff066cfca3c6f90

Comment 3

[James Graham [:jgraham]]

Oh and another blocker on Mac is that after some failures we get compareCanvases claim that there are 14px of differences between actually identical images. We currently work around that by restating after affected tests, but that doesn't scale.

Comment 4

[Mats Palmgren (inactive)]

(In reply to James Graham [:jgraham] from comment #2)
> Yes, missing Ahem on Mac is an issue, but arguably not a blocker since it
> "just" means that tests using Ahem as a non-webfont give an incorrect result
> on mac (of course we should fix that but it's possible to work around by
> using webfonts or similar).

Fwiw, I'm opposed to changing tests that use Ahem to load it as a webfont.
Doing that will make the test take a different code path and we've had
both layout bugs and crash bugs(!) in the past that only occurred when
the test was using a webfont (and vice versa).

The correct solution is to require that Ahem is installed as a system font
on the machines where we run those tests.

Comment 5

[Mats Palmgren (inactive)]

Installing the Ahem font is actually a requirement by WPT:
https://web-platform-tests.org/writing-tests/ahem.html

Comment 6

[James Graham [:jgraham]]

I agree that not using webfonts always covers mode codepaths, but aiui gecko has traditionally used webfonts everywhere rather than installing fonts so I assumed we were OK with the tradeoffs. If we aren't actually then I will change my poisiton that webfonts should be preferred for new tests.

Installing Ahem is indeed a requirement, and on non-Mac platforms we do that. On Mac we *try* to do that, but for some reason it doesn't work (the font added to ~/.fonts doesn't end up in the system font list). If you want to try locally ./mach wpt --install-fonts <test> on a Mac should reproduce the problem. It may be that the only solution is to update the Macs so that Ahem is already installed, but I'd need help from the people running those instances to do that. It also has a few disdvantages since it wouldn't respond to changes in the font file, and wouldn't help people running the tests locally.

Comment 7

[James Graham [:jgraham]]

https://github.com/web-platform-tests/wpt/issues/9105 is an upstream issue about whether to use Ahem as a webfont or not.

Comment 8

[Mats Palmgren (inactive)]

> On Mac we *try* to do that, but for some reason it doesn't work

Could you file a separate bug on that, if there isn't one already?
Please need-info :jfkthame on that bug, I'm pretty sure he can help.

> https://github.com/web-platform-tests/wpt/issues/9105 is an upstream issue

Thanks, I will add a comment there.

Comment 9

[James Graham [:jgraham]]

That bug is bug 1425698 Apparently I had forgotten that it *doesn't* reproduce locally i.e. things work OK with --install-fonts except on CI (but the people running wpt.fyi on Safari in SauceLabs machines had the same kind of problem).

Comment 10

[James Graham [:jgraham]]

Attachment: Bug 1481723 - Enable wpt css tests on Windows,

MozReview-Commit-ID: ekP5TOSaRM

Depends on D7200

Comment 11

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 9012948 [details]
Bug 1481723 - Enable wpt css tests on Windows,

Andreas Tolfsen ❲:ato❳ has approved the revision.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d4de0eb65962
Enable wpt css tests on Windows, r=ato

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/d4de0eb65962