Closed Bug 1499003 Opened 4 years ago Closed 3 years ago

Green up fennec wpt results.


(Testing :: web-platform-tests, enhancement)

Version 3
Not set


(Not tracked)



(Reporter: KWierso, Assigned: KWierso)




(2 files, 1 obsolete file)

Now that bug 1496627 lets us run the android wpt tests locally, and Bug 1495430 (and friends) added support to run them in CI, we need to get them greened up to a point where they can be turned on to run in CI per push, and developers can trust the results to know if new patches break them.

Splitting the tests into several chunks, only a handful of tests are failing per chunk, and a few chunks actually apper to be fully passing:,busted,exception,success,retry,running,pending,runnable&tier=3&tochange=955a7fd1a807357faa79336b16418c41541be24e&fromchange=0ce62298c460df7dccc0d074c73999f13685b11b&group_state=expanded

At first glance, some failures seem to be harness failures (we'll need to fix these), some are timeouts (emulator too slow for the default timeout values, we can maybe just increase the timeout for mobile tests), and some are actually unexpected passes (Fennec supports some webkit prefixed CSS, IIRC, making us pass those tests unexpectedly).

So far all of these tests (ci and local) have been done exclusively with the x86 android emulator, which is much faster than the various ARM emulators. This greening up work will probably be only done with the x86 emulator. It's unclear to me what the plan is for running these tests on the ARM emulators, but I'd hope the only real difference is that we'd need to bump the default timeout value even higher. I'm not planning on dealing with that in this bug, though.
For the timeout value the thing to do is to set the timeout_multiplier value in [1]. We currently just use the function for desktop firefox, which may be conservative for mobile, even on x86.

The first thing to do however might be to just update the expectation values for mobile and see how it looks. There's a script to fetch logs from treeherder in testing/web-platform/update/ although I use the Rust application at [2]; with the latter you should be able to run something like

fetchlogs try 955a7fd1a807357faa79336b16418c41541be24e --out-dir logs --log-type wptreport

And then to set the expectation data to match the results, do something like

./mach wpt-update logs/*

Although that will probably overwrite the default values in some cases (because without non-mobile results it can't understand that these are android-specific). So at some point we might need a try run with all the platforms.

And to support reftests you might just be able to import the MarionetteReftestRunner in and edit the dictionary at [3] to add {"reftest": MarionetteReftestRunner} or something.

That's most of the wpt directories...

There are still a few directories that could use bugs:

These ones (and maybe others) seem to be timing out intermittently:

I guess they're right up against the timeout threshold, and if we bumped that a bit they might more consistently pass?

These ones have lots of failures within them, and could probably be split up further to be more manageable:
Some of these are probably just platform differences (eg device orientation matters on a phone but less so on the desktop) while others look like actual failures. Rather than working through them all now, just disable the ones that fail, so we can maybe turn the fennec wpt tests on while fixing and re-enabling these skipped tests.
Attachment #9024568 - Attachment is obsolete: true
For what it's worth, the wasm abort test should have been fixed properly by bug 1505632 which landed a few days ago. Can it be removed from the exclusion list?
Flags: needinfo?(wkocher)
Pushed by
Update Fennec wpt expectations r=jgraham
Pushed by
Skip some especially flaky fennec wpt tests r=jgraham
Created web-platform-tests PR for changes under testing/web-platform/tests
(In reply to Pulsebot from comment #13)
> Pushed by
> Update annotation references for content-security-policy/navigate-to/ r=me
> a=testonly

This got backed out because it included unrelated changes, and it had the wrong bug number attached to it. Oops.

(In reply to Benjamin Bouvier [:bbouvier] from comment #7)
> For what it's worth, the wasm abort test should have been fixed properly by
> bug 1505632 which landed a few days ago. Can it be removed from the
> exclusion list?

Those patches landed before I could try addressing this, can you file a followup bug for that change?
Flags: needinfo?(wkocher)
Backout by
Backed out changeset 8d5c855aedff per developer's backout on IRC on a CLOSED TREE
Pushed by
Update the referenced bug for skipped fennec wpt user-timing test r=me a=test-only
Pushed by
Update the referenced bug for skipped fennec wpt service-worker tests r=me a=testonly

The leave-open keyword is there and there is no activity for 6 months.
:KWierso, maybe it's time to close this bug?

Flags: needinfo?(wkocher)

All future work is going to be against geckoview, not fennec. I just closed out all of this bug's dependencies, because we're likely to stop running wpt against fennec builds in the near future, so there's nothing left to fix here.

Closed: 3 years ago
Flags: needinfo?(wkocher)
Keywords: leave-open
Resolution: --- → WONTFIX
See Also: → 1550895
Summary: Green up android wpt results. → Green up fennec wpt results.
You need to log in before you can comment on or make changes to this bug.