1725245 - Add Ubuntu 22.04 as a test platform to run tests for MOZ_ENABLE_WAYLAND=1

Status: RESOLVED FIXED

(Testing :: CI Configuration, task)

Description

[Andrew Osmond [:aosmond] (he/him)]

The latest Fedora and Ubuntu releases, and presumably any derivatives thereof, ship Firefox on Wayland by default. They don't use the compatibility layer XWayland, but rather actual Wayland using our pref MOZ_ENABLE_WAYLAND to force it on.

Right now we are unable to test this configuration in CI because our version of Ubuntu is too old to work well as Wayland development is quite active.

We don't need our existing tests on Linux to be fully ported, but it would be good to have the option to run some tests using Wayland on a newer Ubuntu, such as 21.04. This would allow us to start greening the tree for Wayland, and push it out officially from our side.

Comment 1

[Martin Stránský [:stransky] (ni? me)]

The recent Wayland test suite (Bug 1578640) based on Ubuntu 18.04 is working relatively well.

The only issue there is missing focus support. The failures you see there are caused broken focus where we wait for focus (that produces timeouts) or we expect a window has focus (that produces reftest failures).

These failures are pretty random as it depends how the focus is handled by compositor - we use a workaround where we hide/show a window and hope it gets the focus after show.

Only reliable way how to fix that is XDG activation protocol implementation (https://wayland.app/protocols/xdg-activation-v1).
This is not finished yet (https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1845). Robert, do you know when this may be available in Mutter? For instance I don't see it in mutter-40.3-1.fc34.x86_64 package.

So our goal is to have compositor with XDG activation protocol available to test Wayland reliably.

Comment 2

[Robert Mader [:rmader]]

Copying my answer from the issue here:

Current status is that we'll ship it in 41 - we may backport it to 40, but Ubuntu 21.04 still uses 3.38. So I currently don't see to get it into the FF CI apart from

• waiting for 21.10
• using another distro
• asking Ubuntu devs to backport it to Gnome 3.36 so we can use Ubuntu 20.04 (I don't think they'll backport to Ubuntu 21.04, but that would also be an option)

That being said, I think in the Mutter version in Ubuntu 18.04 are quite a few Wayland related bugs - not to mention all those fixes for compositor integration. So I guess we'd want at least Ubuntu 20.04 so we don't run into already fixed issues.

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

I think we want 21.04 given that is the release users are getting Wayland today. Test what is already shipping :). They don't get it in 20.04 by default.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

:stranksy - do you know of prior art/docs on getting wayland running on docker? Currently we run tests on docker with xvfb - my understanding is wayland on xvfb == xwayland; that is not wayland proper. Any advice you have on this would be helpful.

Comment 5

[Robert Mader [:rmader]]

For the record from matrix:

robert.mader
jmaher: for the record, "wayland on xvfb == xwayland" sounds wrong to me. Xwayland runs on top of Wayland, so Xwayland would imply "x11 on wayland on xvfb". So if a nested Wayland session runs on top of xvfb, that's entirely valid.
"Wayland on native backend" would certainly be better than "Wayland on X11 backend", but for docker tests the later should be fine for now.
jmaher
robert.mader: ok, then maybe this isn't such a hard problem to solve- I was reading up on it and kept running into more resources about pure Wayland without xvfb; I will keep looking into this a bit more
robert.mader
jmaher: Gnome-Shell/Mutter only supports native headless Wayland mode since very recently (since Gnome 40), see https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1698. Ubuntu 21.04 still ships 3.38 (because of big UI changes, also mind the versioning switch). So IIUC this will only work from Ubuntu 21.10 on :/
jmaher
ok, that seems very new- probably not what we want right now

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

for unrelated reasons (OOM, should be fixed with upgraded MESA) we would like to run the marionette tests on this new 21.04 image as well

Comment 7

[Martin Stránský [:stransky] (ni? me)]

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #4)

:stranksy - do you know of prior art/docs on getting wayland running on docker? Currently we run tests on docker with xvfb - my understanding is wayland on xvfb == xwayland; that is not wayland proper. Any advice you have on this would be helpful.

The testing wayland scripts are already there - see Bug 1578640

The important part is here:
https://searchfox.org/mozilla-central/rev/28c8f793f9f8c80de24a179b4472098692a7e9a4/taskcluster/scripts/tester/test-linux.sh#210

This launch Wayland compositor (Mutter in this case) which provides Wayland display.

xvfb is used as underlying X11 server as AFAIK Mutter fails to run without X11. If any new version does not need X11 to run, you can just run mutter without it.

Generally when you set WAYLAND_DISPLAY env variable and MOZ_ENABLE_WAYLAND=1 env variable Firefox will run under Wayland and it does not matter if there's X11, Xwayland or any other X server running.

Comment 8

[Robert Mader [:rmader]]

Given that 21.04 is going EOL next week and 22.04 (LTS) is already in alpha, I'm taking the freedom to update the request.

Comment 9

[Martin Stránský [:stransky] (ni? me)]

Ubuntu 22.04 is live now and it's even running Firefox on Wayland by default. We need to do the testing ASAP.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

we just did a manual QA pass on this, ASAP == DONE :)

Comment 11

[Emilio Cobos Álvarez (:emilio)]

But we still don't have CI for this right? That seems unfortunate :(

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

These comments are not productive, lets work to keep our bugzilla comments about issues and solving problems, not opinions.

Comment 13

[Emilio Cobos Álvarez (:emilio)]

I was just trying to confirm that CI for this is not set up yet, sorry if it came across the wrong way... If there's anything I can help with to get it set up let me know, I'd be happy to help.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, CI is not setup, and probably won't be until the summer. We don't have headcount and our contractor is blocked on access (which goes to headcount in other teams). There is a long list of things, our priorities are migrating from AWS to GCP and replacing old performance hardware (old phones and laptops). We are using manual QA to give us sanity checks in the meantime.

The biggest problem we have is that our current images run in docker images and the last 3 times we have spent 3+ months just getting a desktop windowing environment to work properly from within docker. I have proposed switching to a VM where this is a task that can complete in a day or two for a simple experiment, but for automation requires some retooling and provisioning changes (then we are back migrating cloud providers).

This is a higher priority than windows 11. OSX upgrades, or android emulator upgrades.

Comment 15

[Emilio Cobos Álvarez (:emilio)]

Thanks for the update Joel (and for all the work you and your team do)

Comment 17

[Olivier Tilloy]

(In reply to Martin Stránský [:stransky] (ni? me) from comment #9)

Ubuntu 22.04 is live now and it's even running Firefox on Wayland by default. We need to do the testing ASAP.

A minor clarification: Ubuntu 22.04 hasn't been released yet, it will go live on 2022-04-21. Also note that users of Ubuntu 20.04 won't be offered to upgrade until 22.04.1 is released, on 2022-08-04. See the release schedule for details.

Comment 18

[Olivier Tilloy]

And to further mitigate potential issues with Firefox on Wayland not being ready for prime time, we can easily disable it (by setting DISABLE_WAYLAND: 1 in the launch environment) until it is deemed ready.

Comment 19

[Olivier Tilloy]

Done here: https://git.launchpad.net/~mozilla-snaps/+git/firefox-snap/commit/?id=4f3ad5cdc390fedab5db6e2d9a062a2c3bd88474

Comment 20

[Olivier Tilloy]

I've had the firefox snap crash 3 times today, while running it from the candidate channel (100.0-1) in a wayland session (Ubuntu 22.04). These crashes didn't trigger the crash reporter.
The last time (just now) I happened to be running it from a terminal, and I got this:

(firefox:80147): Gdk-WARNING **: 21:44:48.720: The program 'firefox' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadValue'.
  (Details: serial 980422 error_code 2 request_code 53 (core protocol) minor_code 0)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the GDK_SYNCHRONIZE environment
   variable to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

I wonder whether that might speak in favour of reverting to native wayland (which I've been running for months without seeing any such crash)?

Comment 21

[Pascal Chevrel:pascalc]

(In reply to Olivier Tilloy from comment #20)

I've had the firefox snap crash 3 times today, while running it from the candidate channel (100.0-1) in a wayland session (Ubuntu 22.04). These crashes didn't trigger the crash reporter.
The last time (just now) I happened to be running it from a terminal, and I got this:

(firefox:80147): Gdk-WARNING **: 21:44:48.720: The program 'firefox' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadValue'.
  (Details: serial 980422 error_code 2 request_code 53 (core protocol) minor_code 0)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the GDK_SYNCHRONIZE environment
   variable to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

I wonder whether that might speak in favour of reverting to native wayland (which I've been running for months without seeing any such crash)?

Did you file a bug?

Comment 22

[Olivier Tilloy]

(In reply to Pascal Chevrel:pascalc from comment #21)

Did you file a bug?

Now I have: bug 1766849

Comment 23

[Olivier Tilloy]

Another data point: several users have reported that a cold startup of the firefox snap (just after booting the machine) on Ubuntu 22.04 in a Wayland session is significantly faster when using the beta channel than with the candidate channel. The only meaningful difference between the two is that the snap in the candidate channel forces the use of XWayland, whereas beta makes use of native Wayland support.

Comment 24

[Tim Richardson]

in lieu of CI infrastructure, can you accept as an alternative some kind of community feedback? Because Firefox under wayland has been great on Fedora for > 12 months, and the beta channel seems to work, and using xwayland is exposing people to https://gitlab.gnome.org/GNOME/gtk/-/issues/4136 (a gtk4 bug that is triggered by xwayland). While it's not a mozilla problem and I can't speak for the Ubuntu release team, it's a bit ironic that despite efforts to avoid gtk4 bugs in Ubuntu 22.04, the Firefox snap has shipped one anyway.

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

When we upgrade our CI from 18.04 to 22.04 it would be great if we could take care of properly setting up the font cache to not trigger a generation for the very first start of Firefox. For all of our CI jobs it adds another 15s and also some test jobs fail under such a condition because the startup of Firefox takes longer than usual. I'll add bug 1670290 as dependency. Thanks.

Comment 26

[Mike Kaply [:mkaply]]

Where do we stand on this? I thought we were targeting October originally.

Comment 27

[Marco Castelluccio [:marco]]

David is working on standing up a VM, but run into some issues lately.

Comment 28

[Martin Stránský [:stransky] (ni? me)]

Folks, how do we look here? Ubuntu 18.04 (recently used for testing) is EOL and latest nightly can't be build there due to old python 3.6. So all test fixes/tweaks has to be done on try directly which wastes Mozilla resources (right now on Bug 1787182 for instance).

Comment 29

[Gian-Carlo Pascutto [:gcp]]

Michelle, can you provide an update here or link the bugs where that work is happening?

Comment 30

[Mike Hommey [:glandium]]

Maybe bug 1801347?

(In reply to Martin Stránský [:stransky] (ni? me) from comment #28)

Folks, how do we look here? Ubuntu 18.04 (recently used for testing) is EOL and latest nightly can't be build there due to old python 3.6. So all test fixes/tweaks has to be done on try directly which wastes Mozilla resources (right now on Bug 1787182 for instance).

FWIW, 18.04 has python 3.7: https://packages.ubuntu.com/bionic/python3.7

Comment 31

[Martin Stránský [:stransky] (ni? me)]

(In reply to Mike Hommey [:glandium] from comment #30)

Maybe bug 1801347?

Wayland is just a part of it.

AFAIK we need new testing environment to test what we already ship on X11 - EGL, dmabuf/WebGL, disabled system titlebar etc. All "new" features are off and we don't test them due to old setup in Mozilla automation.

Comment 32

[Michelle Goossens [:masterwayz]]

https://mozilla-hub.atlassian.net/browse/RELOPS-376
https://mozilla-hub.atlassian.net/browse/RELOPS-132 (perhaps?)

Now to give a little context, I was assigned to this. Then someone left, and I do all their Mac tasks now so I haven't looked at it (I think this is also why the priority was dropped.)
Julia is on PTO for a few days, so I can't ask her right now what the plan is.
I'll ping-pong the NI over to the main Linux magician at our team.

Comment 33

[Andrew Erickson [:aerickson]]

This is on our radar. We're discussing in Slack #wayland-support.

Comment 34

[Robert Mader [:rmader]]

IIUC there are now several runners using Ubuntu 22.04, notably for bug 1813588, so I guess this can be closed and most dependent bugs can depend on that bug instead?

Comment 35

[Julien Cristau [:jcristau]]

Comment 34 makes sense to me. Andrew, OK to close?

Comment 36

[Andrew Erickson [:aerickson]]

<edited> wrong andrew :)

Comment 37

[Andrew Halberstadt [:ahal]]

Wfm!