Status

enhancement
RESOLVED WONTFIX
8 months ago
4 days ago

People

(Reporter: KWierso, Assigned: KWierso)

Tracking

Version 3
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

Assignee

Description

8 months ago
Now that bug 1496627 lets us run the android wpt tests locally, and Bug 1495430 (and friends) added support to run them in CI, we need to get them greened up to a point where they can be turned on to run in CI per push, and developers can trust the results to know if new patches break them.

Splitting the tests into several chunks, only a handful of tests are failing per chunk, and a few chunks actually apper to be fully passing: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed,busted,exception,success,retry,running,pending,runnable&tier=3&tochange=955a7fd1a807357faa79336b16418c41541be24e&fromchange=0ce62298c460df7dccc0d074c73999f13685b11b&group_state=expanded

At first glance, some failures seem to be harness failures (we'll need to fix these), some are timeouts (emulator too slow for the default timeout values, we can maybe just increase the timeout for mobile tests), and some are actually unexpected passes (Fennec supports some webkit prefixed CSS, IIRC, making us pass those tests unexpectedly).

So far all of these tests (ci and local) have been done exclusively with the x86 android emulator, which is much faster than the various ARM emulators. This greening up work will probably be only done with the x86 emulator. It's unclear to me what the plan is for running these tests on the ARM emulators, but I'd hope the only real difference is that we'd need to bump the default timeout value even higher. I'm not planning on dealing with that in this bug, though.
For the timeout value the thing to do is to set the timeout_multiplier value in [1]. We currently just use the function for desktop firefox, which may be conservative for mobile, even on x86.

The first thing to do however might be to just update the expectation values for mobile and see how it looks. There's a script to fetch logs from treeherder in testing/web-platform/update/fetchlogs.py although I use the Rust application at [2]; with the latter you should be able to run something like

fetchlogs try 955a7fd1a807357faa79336b16418c41541be24e --out-dir logs --log-type wptreport

And then to set the expectation data to match the results, do something like

./mach wpt-update logs/*

Although that will probably overwrite the default values in some cases (because without non-mobile results it can't understand that these are android-specific). So at some point we might need a try run with all the platforms.

And to support reftests you might just be able to import the MarionetteReftestRunner in fennec.py and edit the dictionary at [3] to add {"reftest": MarionetteReftestRunner} or something.

[1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#102
[2] https://github.com/jgraham/fetchlogs
[3] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/fennec.py#27
Assignee

Updated

8 months ago
Keywords: leave-open
Assignee

Updated

8 months ago
Depends on: 1499866
Assignee

Updated

8 months ago
Depends on: 1499871
Assignee

Updated

8 months ago
Depends on: 1499889
Assignee

Updated

8 months ago
Depends on: 1499893
Assignee

Updated

8 months ago
Depends on: 1499900
Assignee

Updated

8 months ago
Depends on: 1499904
Assignee

Updated

8 months ago
Depends on: 1499913
Assignee

Updated

8 months ago
Depends on: 1499945
Assignee

Updated

8 months ago
Depends on: 1499946
Assignee

Updated

8 months ago
Depends on: 1499948
Assignee

Updated

8 months ago
Depends on: 1499951
Assignee

Updated

8 months ago
Depends on: 1499952
Assignee

Updated

8 months ago
Depends on: 1499955
Assignee

Updated

8 months ago
Depends on: 1499957
Assignee

Updated

8 months ago
Depends on: 1499958
Assignee

Updated

8 months ago
Depends on: 1499959
Assignee

Updated

8 months ago
Depends on: 1499962
Assignee

Updated

8 months ago
Depends on: 1499963
Assignee

Updated

8 months ago
Depends on: 1499966
Assignee

Updated

8 months ago
Depends on: 1499967
Assignee

Updated

8 months ago
Depends on: 1499968
Assignee

Updated

8 months ago
Depends on: 1499969
Assignee

Updated

8 months ago
Depends on: 1499972
Assignee

Updated

8 months ago
Depends on: 1499973
Assignee

Updated

8 months ago
Depends on: 1499976
Assignee

Updated

8 months ago
Depends on: 1499978
Assignee

Updated

8 months ago
Depends on: 1499979
Assignee

Updated

8 months ago
Depends on: 1499982
Assignee

Updated

8 months ago
Depends on: 1499983
Assignee

Updated

8 months ago
Depends on: 1499989
Assignee

Updated

8 months ago
Depends on: 1499993
Assignee

Comment 2

8 months ago
That's most of the wpt directories...



There are still a few directories that could use bugs:

These ones (and maybe others) seem to be timing out intermittently:
payment-request
picture-in-picture

I guess they're right up against the timeout threshold, and if we bumped that a bit they might more consistently pass?




These ones have lots of failures within them, and could probably be split up further to be more manageable:
css
encoding
html
intersection-observer
Assignee

Updated

8 months ago
Depends on: 1501536
Assignee

Updated

8 months ago
Depends on: 1501547
Assignee

Updated

8 months ago
Depends on: 1501548
Assignee

Updated

8 months ago
Depends on: 1501549
Assignee

Updated

8 months ago
Depends on: 1501553
Assignee

Updated

8 months ago
Depends on: 1502110
Assignee

Updated

8 months ago
Depends on: 1502202
Assignee

Updated

8 months ago
Depends on: 1502208
Assignee

Updated

8 months ago
Depends on: 1502216
Assignee

Updated

8 months ago
Depends on: 1502225
Assignee

Updated

8 months ago
Depends on: 1503068
Assignee

Updated

8 months ago
Depends on: 1504377
Assignee

Updated

8 months ago
Depends on: 1504487
Assignee

Updated

7 months ago
Depends on: 1504606
Assignee

Comment 3

7 months ago
Some of these are probably just platform differences (eg device orientation matters on a phone but less so on the desktop) while others look like actual failures. Rather than working through them all now, just disable the ones that fail, so we can maybe turn the fennec wpt tests on while fixing and re-enabling these skipped tests.
Attachment #9024568 - Attachment is obsolete: true
For what it's worth, the wasm abort test should have been fixed properly by bug 1505632 which landed a few days ago. Can it be removed from the exclusion list?
Flags: needinfo?(wkocher)

Comment 8

7 months ago
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/268c6832d2be
Update Fennec wpt expectations r=jgraham

Comment 9

7 months ago
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/cda49d66b3cc
Skip some especially flaky fennec wpt tests r=jgraham
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14263 for changes under testing/web-platform/tests
Upstream PR merged
Comment hidden (obsolete)
Assignee

Comment 14

7 months ago
(In reply to Pulsebot from comment #13)
> Pushed by wkocher@mozilla.com:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/8d5c855aedff
> Update annotation references for content-security-policy/navigate-to/ r=me
> a=testonly

This got backed out because it included unrelated changes, and it had the wrong bug number attached to it. Oops.

(In reply to Benjamin Bouvier [:bbouvier] from comment #7)
> For what it's worth, the wasm abort test should have been fixed properly by
> bug 1505632 which landed a few days ago. Can it be removed from the
> exclusion list?

Those patches landed before I could try addressing this, can you file a followup bug for that change?
Flags: needinfo?(wkocher)

Comment 15

7 months ago
Backout by shindli@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/431c208f6787
Backed out changeset 8d5c855aedff per developer's backout on IRC on a CLOSED TREE

Comment 16

6 months ago
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1233b674e1cd
Update the referenced bug for skipped fennec wpt user-timing test r=me a=test-only

Comment 17

6 months ago
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b5f5accfcae
Update the referenced bug for skipped fennec wpt service-worker tests r=me a=testonly
Assignee

Updated

6 months ago
Depends on: 1512535
Assignee

Updated

6 months ago
Depends on: 1512536

The leave-open keyword is there and there is no activity for 6 months.
:KWierso, maybe it's time to close this bug?

Flags: needinfo?(wkocher)
Assignee

Comment 20

4 days ago

All future work is going to be against geckoview, not fennec. I just closed out all of this bug's dependencies, because we're likely to stop running wpt against fennec builds in the near future, so there's nothing left to fix here.

Status: NEW → RESOLVED
Closed: 4 days ago
Flags: needinfo?(wkocher)
Keywords: leave-open
Resolution: --- → WONTFIX
See Also: → wpt-geckoview
Summary: Green up android wpt results. → Green up fennec wpt results.
You need to log in before you can comment on or make changes to this bug.