Open Bug 1578640 (wayland-tests) Opened 5 years ago Updated 1 month ago

[meta] Wayland backend testing

Categories

(Core :: Widget: Gtk, task, P2)

task

Tracking

()

People

(Reporter: stransky, Unassigned)

References

(Depends on 13 open bugs, Blocks 1 open bug)

Details

(Keywords: meta)

Recently I see various issues when running mozilla tests on Firefox/Wayland. Also we need to somehow enable wayland backed on mozilla try tests.

Looks like there's problem with focus:
19:22.18 INFO Error: Unable to restore focus, expect failures and timeouts.

https://searchfox.org/mozilla-central/rev/153feabebc2d13bb4c29ef8adf104ec1ebd246ae/testing/mochitest/tests/SimpleTest/TestRunner.js#326

No longer blocks: 1657289
Depends on: 1657289

I managed to run it after all, there's a first try run on Wayland backend with some random tests selected:
https://treeherder.mozilla.org/jobs?repo=try&revision=2a4d6b19d3cfce3dea771d327872df38d8da906e&selectedTaskRun=H9L5ZIltScC_Wo8TEuY33A.0

Alias: wayland-tests

With some tweaks I'm able to run mochitest locally. The dom directory looks pretty good:

mochitest-browser

Ran 19618 checks (19378 subtests, 240 tests)
Expected results: 19569
Skipped: 42 tests
Unexpected results: 7
subtest: 7 (7 fail)

mochitest-chrome

Ran 25235 checks (24880 subtests, 355 tests)
Expected results: 25206
Skipped: 18 tests
Unexpected results: 10
subtest: 10 (10 fail)

mochitest-plain

Ran 250114 checks (245268 subtests, 4846 tests)
Expected results: 249848
Skipped: 236 tests
Unexpected results: 21
subtest: 21 (20 fail, 1 pass)

There are particular failing tests:

mochitest-browser

dom/credentialmanagement/tests/browser/browser_active_document.js
dom/events/test/browser_mouse_enterleave_switch_tab.js
dom/html/test/browser_fullscreen-contextmenu-esc.js
dom/html/test/browser_fullscreen-newtab.js

mochitest-chrome

dom/tests/mochitest/chrome/test_clipboard_events_chrome.html
dom/tests/mochitest/chrome/test_queryCaretRect.html
dom/tests/mochitest/general/test_innerScreen.xhtml
dom/media/test/chrome/test_accumulated_play_time.html

mochitest-plain

dom/events/test/test_bug422132.html
dom/events/test/pointerevents/test_getCoalescedEvents.html
dom/html/test/test_fullscreen-api.html
dom/svg/test/test_bounds.html
dom/svg/test/test_getCTM.html
dom/tests/mochitest/bugs/test_sizetocontent_clamp.html
dom/tests/mochitest/general/test_interfaces_secureContext.html
dom/tests/mochitest/pointerlock/test_pointerlock-api.html
dom/media/test/test_eme_pssh_in_moof.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelay.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayTCP.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATSrflx.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNoisyUDPBlock.html
dom/media/webrtc/tests/mochitests/test_peerConnection_stats_relayProtocol.html
dom/media/webrtc/tests/mochitests/test_getUserMedia_audioConstraints.html
dom/canvas/test/webgl-mochitest/ensure-exts/test_EXT_texture_filter_anisotropic.html

Andres, is it possible to set a pref key to run the tests? Something like the --env we use to turn on Wayland. We'd need to set widget.wayland.test-workarounds.enabled to true for the testsuite.
Thanks.

Flags: needinfo?(ahal)

Looks like we can run some Wayland tests even on such old infrastructure:

https://treeherder.mozilla.org/jobs?repo=try&revision=46391665343ef368d36cfaca067ce67e7f3cd703&selectedTaskRun=UiPFyOfgRjWz8oNXF4ELUg.0

I'll look at it to have at least some non-failing tests there which we can use for basic testing.

Looks like we can run some Wayland tests even on such old infrastructure:

Pretty cool! One more thing you could try would be adding disabling the Wayland vsync to widget.wayland.test-workarounds.enabled. The Mutter version in Ubuntu 18.04 is 3.28, which didn't get the patches necessary to make it run. I requested a backport[1], but so far it hasn't happened IIUC. So there is a chance that a couple of tests don't succeed because that Mutter bug.

1: https://bugs.launchpad.net/ubuntu/+source/mutter/+bug/1905330

Not via the command line at the moment (there's a bug on file somewhere), but you could add a user_pref to this file and it should get picked up by all major test harnesses:
https://searchfox.org/mozilla-central/source/testing/profiles/base/user.js

(see other user.js files in the profiles dir if you want to target a specific harness or need an example of the format)

Flags: needinfo?(ahal)

Last try, I added wayland test variants:

https://treeherder.mozilla.org/jobs?repo=try&revision=9adc77fe97f383c57859d932aec989987d8ba918&selectedTaskRun=blRy6KvfQCusoYcSdk4XvQ.0

Andrew, do you think this approach is OK? If so I'll submit patches.
Thanks.

Flags: needinfo?(ahal)

Yes, the approach is correct! I might have a few review requests, but that's better left to phabricator so feel free to submit and I can bring them up there. Thanks!

Flags: needinfo?(ahal)

Also note we can land your variant without actually running it on trunk just yet, to make it easier for folks to run on try without needing to block on greening up the suites.

No longer depends on: 1706008

According to Andrew the Wayland tests can be run as:

$ ./mach try fuzzy --full -q "'wayland"

(when Bug 1708482 lands).

See Also: → 1725245
Summary: [Wayland] Make wayland backend work with mozilla test suite. → [Wayland] Wayland backend testing

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

(In reply to Martin Stránský [:stransky] (ni? me) from comment #14)

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

I managed to bribe Jonas to make another Fedora 35 Mutter update (also because of other patches). So in case you want to try it over the next weeks, it should be ready soon: https://bodhi.fedoraproject.org/updates/FEDORA-2021-339e7bf6ef

Great, Thanks! Will look at it tomorrow.

(In reply to Robert Mader [:rmader] from comment #15)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #14)

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

I managed to bribe Jonas to make another Fedora 35 Mutter update (also because of other patches). So in case you want to try it over the next weeks, it should be ready soon: https://bodhi.fedoraproject.org/updates/FEDORA-2021-339e7bf6ef

xdg-activation & mutter-41.2-2.fc35.x86_64 seems to be working well, I can run the testsuite now.

See Also: → 1752113
No longer blocks: wayland-nightly
Depends on: wayland-nightly
Severity: normal → --
Type: enhancement → task
Depends on: 1725245, 1752113
No longer depends on: wayland-nightly
Keywords: meta
See Also: 1725245, 1752113
Summary: [Wayland] Wayland backend testing → [meta] Wayland backend testing
No longer depends on: 1752113

I'm going to spin it again and run another round of tests on Ubuntu 22.04. Will post details later.

Depends on: 1783818
No longer blocks: 1825171
Depends on: 1825171
Depends on: 1846815
Depends on: 1855305
Depends on: 1856368
Depends on: 1863158
Depends on: 1879835
Depends on: 1898340
You need to log in before you can comment on or make changes to this bug.