1578640 - (wayland-tests) [meta] Wayland backend testing

Status: NEW

(Core :: Widget: Gtk, task, P2)

Description

[Martin Stránský [:stransky] (ni? me)]

Recently I see various issues when running mozilla tests on Firefox/Wayland. Also we need to somehow enable wayland backed on mozilla try tests.

Comment 1

[Martin Stránský [:stransky] (ni? me)]

Looks like there's problem with focus:
19:22.18 INFO Error: Unable to restore focus, expect failures and timeouts.

https://searchfox.org/mozilla-central/rev/153feabebc2d13bb4c29ef8adf104ec1ebd246ae/testing/mochitest/tests/SimpleTest/TestRunner.js#326

Comment 2

[Martin Stránský [:stransky] (ni? me)]

I managed to run it after all, there's a first try run on Wayland backend with some random tests selected:
https://treeherder.mozilla.org/jobs?repo=try&revision=2a4d6b19d3cfce3dea771d327872df38d8da906e&selectedTaskRun=H9L5ZIltScC_Wo8TEuY33A.0

Comment 3

[Martin Stránský [:stransky] (ni? me)]

With some tweaks I'm able to run mochitest locally. The dom directory looks pretty good:

mochitest-browser

Ran 19618 checks (19378 subtests, 240 tests)
Expected results: 19569
Skipped: 42 tests
Unexpected results: 7
subtest: 7 (7 fail)

mochitest-chrome

Ran 25235 checks (24880 subtests, 355 tests)
Expected results: 25206
Skipped: 18 tests
Unexpected results: 10
subtest: 10 (10 fail)

mochitest-plain

Ran 250114 checks (245268 subtests, 4846 tests)
Expected results: 249848
Skipped: 236 tests
Unexpected results: 21
subtest: 21 (20 fail, 1 pass)

Comment 4

[Martin Stránský [:stransky] (ni? me)]

There are particular failing tests:

mochitest-browser

dom/credentialmanagement/tests/browser/browser_active_document.js
dom/events/test/browser_mouse_enterleave_switch_tab.js
dom/html/test/browser_fullscreen-contextmenu-esc.js
dom/html/test/browser_fullscreen-newtab.js

mochitest-chrome

dom/tests/mochitest/chrome/test_clipboard_events_chrome.html
dom/tests/mochitest/chrome/test_queryCaretRect.html
dom/tests/mochitest/general/test_innerScreen.xhtml
dom/media/test/chrome/test_accumulated_play_time.html

mochitest-plain

dom/events/test/test_bug422132.html
dom/events/test/pointerevents/test_getCoalescedEvents.html
dom/html/test/test_fullscreen-api.html
dom/svg/test/test_bounds.html
dom/svg/test/test_getCTM.html
dom/tests/mochitest/bugs/test_sizetocontent_clamp.html
dom/tests/mochitest/general/test_interfaces_secureContext.html
dom/tests/mochitest/pointerlock/test_pointerlock-api.html
dom/media/test/test_eme_pssh_in_moof.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelay.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATRelayTCP.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNATSrflx.html
dom/media/webrtc/tests/mochitests/test_peerConnection_basicAudioNoisyUDPBlock.html
dom/media/webrtc/tests/mochitests/test_peerConnection_stats_relayProtocol.html
dom/media/webrtc/tests/mochitests/test_getUserMedia_audioConstraints.html
dom/canvas/test/webgl-mochitest/ensure-exts/test_EXT_texture_filter_anisotropic.html

Comment 5

[Martin Stránský [:stransky] (ni? me)]

Andres, is it possible to set a pref key to run the tests? Something like the --env we use to turn on Wayland. We'd need to set widget.wayland.test-workarounds.enabled to true for the testsuite.
Thanks.

Comment 6

[Martin Stránský [:stransky] (ni? me)]

Looks like we can run some Wayland tests even on such old infrastructure:

https://treeherder.mozilla.org/jobs?repo=try&revision=46391665343ef368d36cfaca067ce67e7f3cd703&selectedTaskRun=UiPFyOfgRjWz8oNXF4ELUg.0

I'll look at it to have at least some non-failing tests there which we can use for basic testing.

Comment 7

[Robert Mader [:rmader]]

Looks like we can run some Wayland tests even on such old infrastructure:

Pretty cool! One more thing you could try would be adding disabling the Wayland vsync to widget.wayland.test-workarounds.enabled. The Mutter version in Ubuntu 18.04 is 3.28, which didn't get the patches necessary to make it run. I requested a backport[1], but so far it hasn't happened IIUC. So there is a chance that a couple of tests don't succeed because that Mutter bug.

1: https://bugs.launchpad.net/ubuntu/+source/mutter/+bug/1905330

Comment 8

[Martin Stránský [:stransky] (ni? me)]

New try looks much better:
https://treeherder.mozilla.org/jobs?repo=try&revision=91f63487ca15b59fb6f750ab43c619de70a958a1

Comment 9

[Andrew Halberstadt [:ahal]]

Not via the command line at the moment (there's a bug on file somewhere), but you could add a user_pref to this file and it should get picked up by all major test harnesses:
https://searchfox.org/mozilla-central/source/testing/profiles/base/user.js

(see other user.js files in the profiles dir if you want to target a specific harness or need an example of the format)

Comment 10

[Martin Stránský [:stransky] (ni? me)]

Last try, I added wayland test variants:

https://treeherder.mozilla.org/jobs?repo=try&revision=9adc77fe97f383c57859d932aec989987d8ba918&selectedTaskRun=blRy6KvfQCusoYcSdk4XvQ.0

Andrew, do you think this approach is OK? If so I'll submit patches.
Thanks.

Comment 11

[Andrew Halberstadt [:ahal]]

Yes, the approach is correct! I might have a few review requests, but that's better left to phabricator so feel free to submit and I can bring them up there. Thanks!

Comment 12

[Andrew Halberstadt [:ahal]]

Also note we can land your variant without actually running it on trunk just yet, to make it easier for folks to run on try without needing to block on greening up the suites.

Comment 13

[Martin Stránský [:stransky] (ni? me)]

According to Andrew the Wayland tests can be run as:

$ ./mach try fuzzy --full -q "'wayland"

(when Bug 1708482 lands).

Comment 14

[Martin Stránský [:stransky] (ni? me)]

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

Comment 15

[Robert Mader [:rmader]]

(In reply to Martin Stránský [:stransky] (ni? me) from comment #14)

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

I managed to bribe Jonas to make another Fedora 35 Mutter update (also because of other patches). So in case you want to try it over the next weeks, it should be ready soon: https://bodhi.fedoraproject.org/updates/FEDORA-2021-339e7bf6ef

Comment 16

[Martin Stránský [:stransky] (ni? me)]

Great, Thanks! Will look at it tomorrow.

Comment 17

[Martin Stránský [:stransky] (ni? me)]

(In reply to Robert Mader [:rmader] from comment #15)

(In reply to Martin Stránský [:stransky] (ni? me) from comment #14)

btw. the testsuite can be run with latest KWin compositor which supports xdg-activation. Mutter is expected to support it in 41.3 version.

I managed to bribe Jonas to make another Fedora 35 Mutter update (also because of other patches). So in case you want to try it over the next weeks, it should be ready soon: https://bodhi.fedoraproject.org/updates/FEDORA-2021-339e7bf6ef

xdg-activation & mutter-41.2-2.fc35.x86_64 seems to be working well, I can run the testsuite now.

Comment 18

[Martin Stránský [:stransky] (ni? me)]

I'm going to spin it again and run another round of tests on Ubuntu 22.04. Will post details later.