Closed Bug 1499003 Opened 2 years ago Closed 1 year ago

Green up fennec wpt results.

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: KWierso, Assigned: KWierso)

References

Details

Attachments

(2 files, 1 obsolete file)

Now that bug 1496627 lets us run the android wpt tests locally, and Bug 1495430 (and friends) added support to run them in CI, we need to get them greened up to a point where they can be turned on to run in CI per push, and developers can trust the results to know if new patches break them.

Splitting the tests into several chunks, only a handful of tests are failing per chunk, and a few chunks actually apper to be fully passing: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed,busted,exception,success,retry,running,pending,runnable&tier=3&tochange=955a7fd1a807357faa79336b16418c41541be24e&fromchange=0ce62298c460df7dccc0d074c73999f13685b11b&group_state=expanded

At first glance, some failures seem to be harness failures (we'll need to fix these), some are timeouts (emulator too slow for the default timeout values, we can maybe just increase the timeout for mobile tests), and some are actually unexpected passes (Fennec supports some webkit prefixed CSS, IIRC, making us pass those tests unexpectedly).

So far all of these tests (ci and local) have been done exclusively with the x86 android emulator, which is much faster than the various ARM emulators. This greening up work will probably be only done with the x86 emulator. It's unclear to me what the plan is for running these tests on the ARM emulators, but I'd hope the only real difference is that we'd need to bump the default timeout value even higher. I'm not planning on dealing with that in this bug, though.
For the timeout value the thing to do is to set the timeout_multiplier value in [1]. We currently just use the function for desktop firefox, which may be conservative for mobile, even on x86.

The first thing to do however might be to just update the expectation values for mobile and see how it looks. There's a script to fetch logs from treeherder in testing/web-platform/update/fetchlogs.py although I use the Rust application at [2]; with the latter you should be able to run something like

fetchlogs try 955a7fd1a807357faa79336b16418c41541be24e --out-dir logs --log-type wptreport

And then to set the expectation data to match the results, do something like

./mach wpt-update logs/*

Although that will probably overwrite the default values in some cases (because without non-mobile results it can't understand that these are android-specific). So at some point we might need a try run with all the platforms.

And to support reftests you might just be able to import the MarionetteReftestRunner in fennec.py and edit the dictionary at [3] to add {"reftest": MarionetteReftestRunner} or something.

[1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#102
[2] https://github.com/jgraham/fetchlogs
[3] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#27
That's most of the wpt directories...



There are still a few directories that could use bugs:

These ones (and maybe others) seem to be timing out intermittently:
payment-request
picture-in-picture

I guess they're right up against the timeout threshold, and if we bumped that a bit they might more consistently pass?




These ones have lots of failures within them, and could probably be split up further to be more manageable:
css
encoding
html
intersection-observer
Some of these are probably just platform differences (eg device orientation matters on a phone but less so on the desktop) while others look like actual failures. Rather than working through them all now, just disable the ones that fail, so we can maybe turn the fennec wpt tests on while fixing and re-enabling these skipped tests.
Attachment #9024568 - Attachment is obsolete: true
For what it's worth, the wasm abort test should have been fixed properly by bug 1505632 which landed a few days ago. Can it be removed from the exclusion list?
Flags: needinfo?(wkocher)
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/268c6832d2be
Update Fennec wpt expectations r=jgraham
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/cda49d66b3cc
Skip some especially flaky fennec wpt tests r=jgraham
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14263 for changes under testing/web-platform/tests
(In reply to Pulsebot from comment #13)
> Pushed by wkocher@mozilla.com:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/8d5c855aedff
> Update annotation references for content-security-policy/navigate-to/ r=me
> a=testonly

This got backed out because it included unrelated changes, and it had the wrong bug number attached to it. Oops.

(In reply to Benjamin Bouvier [:bbouvier] from comment #7)
> For what it's worth, the wasm abort test should have been fixed properly by
> bug 1505632 which landed a few days ago. Can it be removed from the
> exclusion list?

Those patches landed before I could try addressing this, can you file a followup bug for that change?
Flags: needinfo?(wkocher)
Backout by shindli@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/431c208f6787
Backed out changeset 8d5c855aedff per developer's backout on IRC on a CLOSED TREE
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1233b674e1cd
Update the referenced bug for skipped fennec wpt user-timing test r=me a=test-only
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b5f5accfcae
Update the referenced bug for skipped fennec wpt service-worker tests r=me a=testonly

The leave-open keyword is there and there is no activity for 6 months.
:KWierso, maybe it's time to close this bug?

Flags: needinfo?(wkocher)

All future work is going to be against geckoview, not fennec. I just closed out all of this bug's dependencies, because we're likely to stop running wpt against fennec builds in the near future, so there's nothing left to fix here.

Status: NEW → RESOLVED
Closed: 1 year ago
Flags: needinfo?(wkocher)
Keywords: leave-open
Resolution: --- → WONTFIX
See Also: → 1550895
Summary: Green up android wpt results. → Green up fennec wpt results.
You need to log in before you can comment on or make changes to this bug.