Closed Bug 1657289 Opened 5 years ago Closed 5 years ago

Add Wayland backed option to Linux try

Categories

(Developer Infrastructure :: Try, enhancement, P3)

Product:
Developer Infrastructure
Component:
Try
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: stransky, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [dev-prod-2020])

Attachments

(1 file)

wayland-test.patch
3.03 KB, patch
Details | Diff | Splinter Review

Please add Wayland Linux backend to automated testing, right now we test only X11 backend. Mutter compositor can be used to run Wayland builds.

If X11 is running, run test suite under Mutter compositor as:

$mutter --wayland --nested &
if [ -z "$WAYLAND_DISPLAY" ]; then
export WAYLAND_DISPLAY=wayland-0
else
export WAYLAND_DISPLAY=wayland-1
fi
$MOZ_ENABLE_WAYLAND=1 ./mach (test, mochitest,....)

If X11 is not running, we need to use xvfb-run to launch X11 first and ten run mutter in it:

xvfb-run mutter --wayland --nested &
if [ -z "$WAYLAND_DISPLAY" ]; then
export WAYLAND_DISPLAY=wayland-0
else
export WAYLAND_DISPLAY=wayland-1
fi
$MOZ_ENABLE_WAYLAND=1 ./mach (test, mochitest,....)

Wayland specific bugs on testsuite are tracked as Bug 1578640.

Depends on: wayland-tests
Component: General → Try
Product: Testing → Firefox Build System
Version: Default → unspecified

Mike, any idea how to proceed here? Would be great to have Wayland backend enabled for at least some subset of tests. The testsuite should work fine, at least from my testing on Fedora.
Thanks.

Flags: needinfo?(mh+mozilla)

I was going to redirect to :egao, but he's on PTO until the 20th. Andrew, what do you think?

Flags: needinfo?(mh+mozilla) → needinfo?(ahal)

Seems like a good idea!

If all we care about is try, we could set it up behind the MOZ_ENABLE_WAYLAND env in this script:
https://searchfox.org/mozilla-central/source/taskcluster/scripts/tester/test-linux.sh

Then you could pass in --env via mach try (or we could add a dedicated flag for it). Though maybe we'd want to make it a dedicated platform that runs some handful of tests as a sanity check?

Unsure of priorities here or when someone will be free to work on it. I'll add it to the backlog and mention it to :ajones at our next 1 on 1.

Flags: needinfo?(ahal)
Whiteboard: [dev-prod-2020]

Forgot that script is run as part of the docker image build, so passing flags at task runtime won't work. We'll likely need official platforms here.

(In reply to Andrew Halberstadt [:ahal] from comment #4)

Seems like a good idea!

If all we care about is try, we could set it up behind the MOZ_ENABLE_WAYLAND env in this script:
https://searchfox.org/mozilla-central/source/taskcluster/scripts/tester/test-linux.sh

Then you could pass in --env via mach try (or we could add a dedicated flag for it). Though maybe we'd want to make it a dedicated platform that runs some handful of tests as a sanity check?

It also needs to run mutter compositor prior of that which provides the Wayland environment. It can be done by mutter,
see comment 0

I'm confused by what "prior to that" means. There is no "prior" to test-linux.sh, at least as far as in-tree configuration goes.

(In reply to Andrew Halberstadt [:ahal] from comment #7)

I'm confused by what "prior to that" means. There is no "prior" to test-linux.sh, at least as far as in-tree configuration goes.

As X11 can be run without actual display by Xvfb (https://en.wikipedia.org/wiki/Xvfb) Wayland also allows that. To run Firefox in Wayland mode you need to provide a Wayland environment for it. It can be done by the mutter compositor, which is run before firefox actually starts.

https://searchfox.org/mozilla-central/source/taskcluster/scripts/tester/test-linux.sh will need a patch to start mutter compositor when MOZ_ENABLE_WAYLAND env variable is set. It's similar to the NEED_XVFB used there.

I'll look at it.

(In reply to Martin Stránský [:stransky] from comment #8)

(In reply to Andrew Halberstadt [:ahal] from comment #7)

I'm confused by what "prior to that" means. There is no "prior" to test-linux.sh, at least as far as in-tree configuration goes.

As X11 can be run without actual display by Xvfb (https://en.wikipedia.org/wiki/Xvfb) Wayland also allows that. To run Firefox in Wayland mode you need to provide a Wayland environment for it. It can be done by the mutter compositor, which is run before firefox actually starts.

https://searchfox.org/mozilla-central/source/taskcluster/scripts/tester/test-linux.sh will need a patch to start mutter compositor when MOZ_ENABLE_WAYLAND env variable is set. It's similar to the NEED_XVFB used there.

I'll look at it.

Got it, I think we're on the same page and I was just tripped up by some wording. Thanks, if you can provide a patch for test-linux.sh it would be a big help. Also apparently run-task does invoke test-linux.sh for mozharness based tests:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/job/mozharness_test.py#188

Which means it's not only run as part of the image build, and we can simply add a runtime flag to enable Wayland (assuming we only care about running this configuration on try).

Thanks, there's a rough patch for it.

Unfortunately I have no idea how to test it but the logic should be clear. You also need to have mutter installed and wayland libraries. I'm not sure about Ubuntu/Debian but Fedora ships:

libwayland-client-1.18.0
libwayland-server-1.18.0
libwayland-egl-1.18.0
libwayland-cursor-1.18.0

Thanks.

(In reply to Andrew Halberstadt [:ahal] from comment #9)

Which means it's not only run as part of the image build, and we can simply add a runtime flag to enable Wayland (assuming we only care about running this configuration on try).

Yes, try is enough and you just need to set MOZ_ENABLE_WAYLAND env variable.
Thanks.

Thanks!

You can test it out by running ./mach try fuzzy --env MOZ_ENABLE_WAYLAND=1. Would you mind submitting the above to phabricator? If passing --env isn't ideal we could add a shortcut (though if this isn't going to be a super common use case, I might just suggest we let people pass --env).

Priority: -- → P3

I'd like to highlight that having the Wayland backend tested by default in the CI is also something we would need soon, if possible. See e.g. https://phabricator.services.mozilla.com/D85496#2669079

We already have very few Wayland devs - using their time on fixing regressions from other peoples patches makes things unnecessary hard :/

Sure, would you mind filing a new bug under Firefox Build System :: Task Configuration that depends on this one? Fyi the CI teams were very heavily hit by the layoffs, so expect requests to be processed slowly.

(If we're going to be creating a new configuration for wayland anyway, I vote we just leave the --env as the only means of enabling this on try for now)

Depends on: 1663159

(In reply to Andrew Halberstadt [:ahal] from comment #14)

Sure, would you mind filing a new bug under Firefox Build System :: Task Configuration that depends on this one? Fyi the CI teams were very heavily hit by the layoffs, so expect requests to be processed slowly.

Filed as Bug 1663159, Thanks. In order to work properly we need to have mutter installed in the environment.

Latest trunk with ./mach try fuzzy --env MOZ_ENABLE_WAYLAND=1 --env GDK_BACKEND=wayland
(we need to set GDK_BACKEND=wayland as Ubuntu 18.04 has too old Wayland implementation which is off by default).

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3a5cfe7a04a480c2f9797c08095800615f412bd0

I wonder if it's possible to add Ubuntu 20.04 to the try test pool to get latest (and perhaps only working) Wayland enabled system.

Depends on: 1665844
Blocks: wayland-tests
No longer depends on: wayland-tests

Release managers repeatedly refuse uplifting bug fixes for Wayland because of this: bug 1672139 comment 27
This is a strange situation: Firefox' Wayland backend is apparently enabled by default on Fedora while it couldn't be enabled on Nightly yet (bug 1543600) due to missing CI (bug 1578640). IMHO it would be important to test a shipped feature.
According to bug 1648872 comment 16, Mozilla builds Wayland support on old Linux: "Debian 8 / GTK 3.14 per bug 1634204"
According to bug 1644312 comment 11, Firefox' Wayland backend needs a Mutter/Kwin version that supports wl_surface_damage_buffer to be able to actually run. Mutter supports it since 3.25.4 which would mean Debian 9 stable must be used instead of Debian 8 oldstable to test the Wayland backend. (Also, Wayland requires Mesa drivers to be used if it's not the case.)

Is the issue about building or actually running the tests? Because the test really should run on something more recent IMHO. Optimal would be something like Ubuntu 20.04 / GS 3.36 - there have been many bugs fixed on the way, in Mutter, GTK, Mesa and probably more packages :/

Waiting for Debian oldstable to catch up with that would cost us several more years I assume.

(In reply to Robert Mader [:rmader] from comment #20)

Is the issue about building or actually running the tests? Because the test really should run on something more recent IMHO. Optimal would be something like Ubuntu 20.04 / GS 3.36 - there have been many bugs fixed on the way, in Mutter, GTK, Mesa and probably more packages :/

Waiting for Debian oldstable to catch up with that would cost us several more years I assume.

Rober, a recent problem is that we can't launch Wayland session at Mozilla automation - mutter refuses to start and complains.
See https://treeherder.mozilla.org/logviewer?job_id=315960005&repo=try&lineNumber=2345

[task 2020-09-17T14:58:37.515Z] (mutter:57): mutter-WARNING **: 14:58:37.515: Can't initialize KMS backend: Could not get session ID: No such file or directory

I have some WIP patches but it's still now working.

Summoning Jonas :)

Jonas, does this ring a bell? Otherwise I'll try to have a look later.

Flags: needinfo?(jadahl)

The real deal here is to run mutter as a wayland compositor without gnome session, if that's even possible. We can set XDG dirs, set up dbus and so on, see the actual script with window manager setup:

https://searchfox.org/mozilla-central/rev/c938c7416c633639a5c8ce4412be586eefb48005/taskcluster/scripts/tester/test-linux.sh#168

Mutter can't run headlessly on top of the native backend right now; to be able to run automated tests (that doesn't require emulated input without) I suggest to do what mutter itself does to test Wayland clients - run the nested backend inside Xvfb. See .gitlab-ci.yml in mutter/ for how.

However, running headlessly without being drm master etc, is planned, and in-progress. With it, things like emulated input (currently via the remote desktop API, eventually via libei) will work.

Flags: needinfo?(jadahl)

To not have mutter try to run on top of KMS, pass --nested in combination with --wayland.

Mutter seems to start in the latest try but Firefox crashes on start...needs some more investigation then:
https://treeherder.mozilla.org/logviewer?job_id=320819955&repo=try&lineNumber=2099

I see, I'm getting these errors:

[task 2020-11-05T11:58:05.265Z] Disabling glamor and dri3, EGL setup failed
[task 2020-11-05T11:58:05.265Z] Failed to initialize glamor, falling back to sw
[task 2020-11-05T11:58:05.313Z] Window manager warning: Failed to set environment variable DISPLAY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist
[task 2020-11-05T11:58:05.316Z] Window manager warning: Failed to set environment variable WAYLAND_DISPLAY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist
[task 2020-11-05T11:58:05.316Z] Window manager error: Unable to open X display :1
[task 2020-11-05T11:58:05.320Z] (EE)
[task 2020-11-05T11:58:05.320Z] Fatal server error:
[task 2020-11-05T11:58:05.320Z] (EE) failed to read Wayland events: Connection reset by peer
[task 2020-11-05T11:58:05.320Z] (EE)

(In reply to Martin Stránský [:stransky] from comment #18)

I wonder if it's possible to add Ubuntu 20.04 to the try test pool to get latest (and perhaps only working) Wayland enabled system.

On the surface you could try modifying the image we use:
https://searchfox.org/mozilla-central/source/taskcluster/docker/ubuntu1804-test/Dockerfile#1

Changing that line should trigger image builds as dependencies of the test tasks. There will likely be further issues though and we wouldn't be able to commit to a green baseline of tests. Feel free to file a bug under Firefox Build System :: Task Configuration anyway.

(In reply to Martin Stránský [:stransky] from comment #29)

I see, I'm getting these errors:

[task 2020-11-05T11:58:05.265Z] Disabling glamor and dri3, EGL setup failed
[task 2020-11-05T11:58:05.265Z] Failed to initialize glamor, falling back to sw
[task 2020-11-05T11:58:05.313Z] Window manager warning: Failed to set environment variable DISPLAY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist
[task 2020-11-05T11:58:05.316Z] Window manager warning: Failed to set environment variable WAYLAND_DISPLAY for gnome-session: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist

These are expected AFAIK - at least I get the same errors on my local machine without them being fatal.

[task 2020-11-05T11:58:05.316Z] Window manager error: Unable to open X display :1

Hm, is display :1 right? Just from a quick look, I'd expect it to be :0. Anyhow, this appears to be part where things fail.

Edit: is NEED_XVFB really set?

Some things that come to mind to double check:

  • Is a dbus session running?
  • Do you need X11 support too? If yes, then make sure Xwayland is available, otherwise pass --no-x11 to the mutter command next to --nested.

I managed to run the script locally on console with some minor modifications so I'll look at it.

Hm, works fine on Fedora 32. I'm going to test that on Ubuntu 18.04 which is installed on Mozilla test boxes.

As Bug 1665844 landed we have basic Wayland support there.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Product: Firefox Build System → Developer Infrastructure
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: