1598754 - Remove w3c-css/received/css-writing-modes reftests

Status: RESOLVED FIXED

(Core :: Layout, task)

Description

[Ting-Yu Lin [:TYLin] (PST, UTC-8)]

Before removing, we need to investigate the pass/fail state is the same as those in the web-platform-tests, or figuring out why they are different.

Bug 1594937 comment 2 is a sample analysis.

Comment 1

[Ting-Yu Lin [:TYLin] (PST, UTC-8)]

Attachment: An analysis to align pass/fail state between received css-writing-modes and wpt reftests.

This patch removes all the random-if and fuzzy-if, and align the fail/pass state with wpt.

Here is the try run.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=48bcaa150f188bd9cb4f44854452d2423eeb11a5

I've add some comments to individual tests in failures.list for the
different results between the received reftests and reftest in wpt. Here
is some additional observation:

- The unexpected fails inline-block-alignment-007.xht was because the
test is modified in wpt in
https://github.com/web-platform-tests/wpt/pull/19059

- The text-combine-upright tests has unexpected pass because the test
contains one 'match' test and one 'mismatch' test. WPT consider them
'fail' of one of them are fail, but our own reftest framework show them
separately.

- Non-webrender OSX (e.g. OS X 10.14 debug) has a lot of failures
because of removing fuzzy annotations. Those fuzzy annotations are
really necessary in our own reftest framework, but not on wpt. I think
they're just noise.

To sum up, I didn't see any major differences that are worth keeping
these received writing-mode reftests.

Comment 2

[Ting-Yu Lin [:TYLin] (PST, UTC-8)]

Attachment: Bug 1598754 Part 1 - Remove w3c-css/received/css-writing-modes reftests.

See bug 1598754 comment 1 for an analysis and observation after aligning
the pass/fail state with wpt.

This patch is generated by

1. Migrate the bug numbers from failures.list to corresponding wpt int files.
2. removing os.path.join("css-writing-mode"), in import-tests.py.
3. removing the relevent annotations of css-writing-mode in failures.list.
4. running import-tests.py on a wpt repository at commit
   15f199c91a72b0d51bf0a12b3b77827ecb5051ff.

Comment 3

[Ting-Yu Lin [:TYLin] (PST, UTC-8)]

Attachment: Bug 1598754 Part 2 - Remove the infrastructure that imports wpt tests.

The last css suite (css-writing-mode) is removed in Part 1. We can
remove the infrastructure.

Depends on D62984

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

• Non-webrender OSX (e.g. OS X 10.14 debug) has a lot of failures
  because of removing fuzzy annotations. Those fuzzy annotations are
  really necessary in our own reftest framework, but not on wpt. I think
  they're just noise.

I at least noticed some when I was looking at the bug number transfers where the reftest harness version had a fuzzy annotation for non-webrender OSX specifically, and the WPT was annotated as fail for all mac. That seems less than optimal. Could the wpt annotation be reduced to being a fuzzy and/or could it be limited to webrender? For example, outline-inline-vrl-006.html.ini, or inline-block-alignment-006.xht.ini .

I didn't search for this exhaustively, though, so I'm curious how this category of changes was deemed ok by the analysis you did above?

Comment 5

[Ting-Yu Lin [:TYLin] (PST, UTC-8)]

Could the wpt annotation be reduced to being a fuzzy and/or could it be limited to webrender?

You're right. I overlook this. It is better to preserve the fuzzy annotation. Also, I just realize we didn't run wpt reftests on macOS with webrender enabled, so it is important to annotate the non-webrender macOS runs to pass with fuzzy.

Here is a try that changes those wpt tests that fail only on macOS to PASS. All of them are fuzzy. So I'll use this as a guidance to update our wpt ini.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=05afc9c90112d5423328fe924443c91750054252

Since Part 1 is too large to display properly in phabricator UI, making the review and writing inline comments difficult. I'll consolidate all the wpt ini modification to Part 0.

Comment 6

[Ting-Yu Lin [:TYLin] (PST, UTC-8)]

Attachment: Bug 1598754 Part 0 - Migrate test annotations from the received CSS reftests to wpt ini files.

In this patch, we migrate the fuzzy annotations and bug numbers from
layout/reftests/w3c-css/failures.list to corresponding wpt ini files.

That is, some of the failures on macOS in wpt ini files are actually
successes with some fuzzy.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/33e4b5e3f51b
Part 0 - Migrate test annotations from the received CSS reftests to wpt ini files. r=dbaron
https://hg.mozilla.org/integration/autoland/rev/b066d8667720
Part 1 - Remove w3c-css/received/css-writing-modes reftests. r=dbaron
https://hg.mozilla.org/integration/autoland/rev/d9a3e282d399
Part 2 - Remove the infrastructure that imports wpt tests. r=dbaron

Comment 8

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/33e4b5e3f51b
https://hg.mozilla.org/mozilla-central/rev/b066d8667720
https://hg.mozilla.org/mozilla-central/rev/d9a3e282d399