Enable CSS tests on Windows/Mac

RESOLVED FIXED in Firefox 64

Status

enhancement
RESOLVED FIXED
11 months ago
8 months ago

People

(Reporter: jgraham, Assigned: jgraham)

Tracking

(Depends on 3 bugs, Blocks 2 bugs)

Version 3
mozilla64
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox64 fixed)

Details

Attachments

(1 attachment)

Assignee

Description

11 months ago
On mac these were originally disabled for capacity reasons, and on Windows for some blank screenshots. However both of these issues are now solved.


A recent try run [1] shows two problems:

* On mac we sometimes get bogus results where after a fail tests are recorded as having pixel differences even though the reftest analyzer shows that they're identical

* On Windows we seem to have a lot of differences due to antialiasing.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a57bb4a7319cde5d8c3c5387ac45f655158b11e0&selectedJob=188953746
(In reply to James Graham [:jgraham] from comment #0)
> On mac these were originally disabled for capacity reasons,

bug 1425698 was an issue for wpt-on-mac, too, right?

> [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=a57bb4a7319cde5d8c3c5387ac45f655158b11e0&selectedJob=188953746

(I was mildly curious to take a look at this Try run, but sadly, it looks like its log files have been purged due to our standard Try expiration processes.)
Assignee

Comment 2

9 months ago
Yes, missing Ahem on Mac is an issue, but arguably not a blocker since it "just" means that tests using Ahem as a non-webfont give an incorrect result on mac (of course we should fix that but it's possible to work around by using webfonts or similar).

A more recent try run where I added all the prefs that are set in the reftest harness but not in wpt, in case that fixed any issues (it didn't):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5b020d7b8b1c8ce716140f39eff066cfca3c6f90
Assignee

Comment 3

9 months ago
Oh and another blocker on Mac is that after some failures we get compareCanvases claim that there are 14px of differences between actually identical images. We currently work around that by restating after affected tests, but that doesn't scale.

Comment 4

9 months ago
(In reply to James Graham [:jgraham] from comment #2)
> Yes, missing Ahem on Mac is an issue, but arguably not a blocker since it
> "just" means that tests using Ahem as a non-webfont give an incorrect result
> on mac (of course we should fix that but it's possible to work around by
> using webfonts or similar).

Fwiw, I'm opposed to changing tests that use Ahem to load it as a webfont.
Doing that will make the test take a different code path and we've had
both layout bugs and crash bugs(!) in the past that only occurred when
the test was using a webfont (and vice versa).

The correct solution is to require that Ahem is installed as a system font
on the machines where we run those tests.

Comment 5

9 months ago
Installing the Ahem font is actually a requirement by WPT:
https://web-platform-tests.org/writing-tests/ahem.html
Assignee

Comment 6

9 months ago
I agree that not using webfonts always covers mode codepaths, but aiui gecko has traditionally used webfonts everywhere rather than installing fonts so I assumed we were OK with the tradeoffs. If we aren't actually then I will change my poisiton that webfonts should be preferred for new tests.

Installing Ahem is indeed a requirement, and on non-Mac platforms we do that. On Mac we *try* to do that, but for some reason it doesn't work (the font added to ~/.fonts doesn't end up in the system font list). If you want to try locally ./mach wpt --install-fonts <test> on a Mac should reproduce the problem. It may be that the only solution is to update the Macs so that Ahem is already installed, but I'd need help from the people running those instances to do that. It also has a few disdvantages since it wouldn't respond to changes in the font file, and wouldn't help people running the tests locally.
Assignee

Comment 7

9 months ago
https://github.com/web-platform-tests/wpt/issues/9105 is an upstream issue about whether to use Ahem as a webfont or not.

Comment 8

9 months ago
> On Mac we *try* to do that, but for some reason it doesn't work

Could you file a separate bug on that, if there isn't one already?
Please need-info :jfkthame on that bug, I'm pretty sure he can help.

> https://github.com/web-platform-tests/wpt/issues/9105 is an upstream issue

Thanks, I will add a comment there.
Assignee

Comment 9

9 months ago
That bug is bug 1425698 Apparently I had forgotten that it *doesn't* reproduce locally i.e. things work OK with --install-fonts except on CI (but the people running wpt.fyi on Safari in SauceLabs machines had the same kind of problem).
Depends on: 1425698
Assignee

Updated

9 months ago
Depends on: 1494715
Assignee

Updated

9 months ago
Depends on: 1490096
Assignee

Updated

9 months ago
Depends on: 1494719
Assignee

Updated

9 months ago
Depends on: 1490272, 1494974, 1494977
Assignee

Comment 10

9 months ago
MozReview-Commit-ID: ekP5TOSaRM

Depends on D7200
Comment on attachment 9012948 [details]
Bug 1481723 - Enable wpt css tests on Windows,

Andreas Tolfsen ❲:ato❳ has approved the revision.
Attachment #9012948 - Flags: review+
Assignee

Updated

9 months ago
Blocks: 1495047
Assignee

Updated

9 months ago
Blocks: 1490969
Attachment #9012948 - Attachment description: Bug 1481723 - Enable wpt css tests on all platforms → Bug 1481723 - Enable wpt css tests on all platforms,
Depends on: 1498698
Attachment #9012948 - Attachment description: Bug 1481723 - Enable wpt css tests on all platforms, → Bug 1481723 - Enable wpt css tests on Windows,

Comment 13

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/d4de0eb65962
Status: NEW → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Assignee: nobody → james
Assignee

Updated

8 months ago
Depends on: 1505739
You need to log in before you can comment on or make changes to this bug.