1499003 - Green up fennec wpt results.

Status: RESOLVED WONTFIX

(Testing :: web-platform-tests, enhancement)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Now that bug 1496627 lets us run the android wpt tests locally, and Bug 1495430 (and friends) added support to run them in CI, we need to get them greened up to a point where they can be turned on to run in CI per push, and developers can trust the results to know if new patches break them.

Splitting the tests into several chunks, only a handful of tests are failing per chunk, and a few chunks actually apper to be fully passing: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed,busted,exception,success,retry,running,pending,runnable&tier=3&tochange=955a7fd1a807357faa79336b16418c41541be24e&fromchange=0ce62298c460df7dccc0d074c73999f13685b11b&group_state=expanded

At first glance, some failures seem to be harness failures (we'll need to fix these), some are timeouts (emulator too slow for the default timeout values, we can maybe just increase the timeout for mobile tests), and some are actually unexpected passes (Fennec supports some webkit prefixed CSS, IIRC, making us pass those tests unexpectedly).

So far all of these tests (ci and local) have been done exclusively with the x86 android emulator, which is much faster than the various ARM emulators. This greening up work will probably be only done with the x86 emulator. It's unclear to me what the plan is for running these tests on the ARM emulators, but I'd hope the only real difference is that we'd need to bump the default timeout value even higher. I'm not planning on dealing with that in this bug, though.

Comment 1

[James Graham [:jgraham]]

For the timeout value the thing to do is to set the timeout_multiplier value in [1]. We currently just use the function for desktop firefox, which may be conservative for mobile, even on x86.

The first thing to do however might be to just update the expectation values for mobile and see how it looks. There's a script to fetch logs from treeherder in testing/web-platform/update/fetchlogs.py although I use the Rust application at [2]; with the latter you should be able to run something like

fetchlogs try 955a7fd1a807357faa79336b16418c41541be24e --out-dir logs --log-type wptreport

And then to set the expectation data to match the results, do something like

./mach wpt-update logs/*

Although that will probably overwrite the default values in some cases (because without non-mobile results it can't understand that these are android-specific). So at some point we might need a try run with all the platforms.

And to support reftests you might just be able to import the MarionetteReftestRunner in fennec.py and edit the dictionary at [3] to add {"reftest": MarionetteReftestRunner} or something.

[1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#102
[2] https://github.com/jgraham/fetchlogs
[3] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#27

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

That's most of the wpt directories...



There are still a few directories that could use bugs:

These ones (and maybe others) seem to be timing out intermittently:
payment-request
picture-in-picture

I guess they're right up against the timeout threshold, and if we bumped that a bit they might more consistently pass?




These ones have lots of failures within them, and could probably be split up further to be more manageable:
css
encoding
html
intersection-observer

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1499003 - Disable on fennec most of the wpt tests that fail on fennec r?jgraham

Some of these are probably just platform differences (eg device orientation matters on a phone but less so on the desktop) while others look like actual failures. Rather than working through them all now, just disable the ones that fail, so we can maybe turn the fennec wpt tests on while fixing and re-enabling these skipped tests.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1499003 - Update Fennec wpt expectations r?jgraham

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1499003 - Skip some especially flaky fennec wpt tests r?jgraham

Depends on D13012

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

With those patches applied, the try run looks like https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&group_state=expanded&revision=64890a86f79fd81da0e0fd4f8c266ab36dad8d86

Comment 7

[Benjamin Bouvier [:bbouvier] (inactive)]

For what it's worth, the wasm abort test should have been fixed properly by bug 1505632 which landed a few days ago. Can it be removed from the exclusion list?

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/268c6832d2be
Update Fennec wpt expectations r=jgraham

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cda49d66b3cc
Skip some especially flaky fennec wpt tests r=jgraham

Comment 10

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/268c6832d2be
https://hg.mozilla.org/mozilla-central/rev/cda49d66b3cc

Comment 11

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14263 for changes under testing/web-platform/tests

Comment 12

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8d5c855aedff
Update annotation references for content-security-policy/navigate-to/ r=me a=testonly

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Pulsebot from comment #13)
> Pushed by [e-mail]:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/8d5c855aedff
> Update annotation references for content-security-policy/navigate-to/ r=me
> a=testonly

This got backed out because it included unrelated changes, and it had the wrong bug number attached to it. Oops.

(In reply to Benjamin Bouvier [:bbouvier] from comment #7)
> For what it's worth, the wasm abort test should have been fixed properly by
> bug 1505632 which landed a few days ago. Can it be removed from the
> exclusion list?

Those patches landed before I could try addressing this, can you file a followup bug for that change?

Comment 15

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/431c208f6787
Backed out changeset 8d5c855aedff per developer's backout on IRC on a CLOSED TREE

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1233b674e1cd
Update the referenced bug for skipped fennec wpt user-timing test r=me a=test-only

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b5f5accfcae
Update the referenced bug for skipped fennec wpt service-worker tests r=me a=testonly

Comment 18

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/1233b674e1cd
https://hg.mozilla.org/mozilla-central/rev/5b5f5accfcae

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:KWierso, maybe it's time to close this bug?

Comment 20

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

All future work is going to be against geckoview, not fennec. I just closed out all of this bug's dependencies, because we're likely to stop running wpt against fennec builds in the near future, so there's nothing left to fix here.