1684365 - Firefox 84.0.1 crashes in Wayland mode

Status: RESOLVED FIXED

(Core :: Widget: Gtk, defect, P3)

Description

[Michael Hofmann]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

I compiled and installed Firefox 48.0.1 under Gentoo Linux.

Actual results:

Firefox works in X11 mode, but it crashes in Wayland mode with SIGSEGV.

I traced down the error to: firefox-84.0.1/toolkit/components/remote/nsRemoteService.cpp:172

For a backtrace and details: See https://bugs.gentoo.org/762035

Expected results:

An error check is missing before line 172. The error check should make sure that initialization succeeded. It should abort gently with an error message if initialization failed.

Below is the function. I marked line 172 and the position of the missing error check.

void nsRemoteService::StartupServer() {
if (mRemoteServer) {
return;
}

if (mProfile.IsEmpty()) {
return;
}

#ifdef MOZ_WIDGET_GTK
bool useX11Remote = GDK_IS_X11_DISPLAY(gdk_display_get_default());

if defined(MOZ_ENABLE_DBUS)

if (!useX11Remote || getenv(DBUS_REMOTE_ENV)) {
mRemoteServer = MakeUnique<nsDBusRemoteServer>();
}

endif

if (!mRemoteServer && useX11Remote) {
mRemoteServer = MakeUnique<nsGTKRemoteServer>();
}
#elif defined(XP_WIN)
mRemoteServer = MakeUnique<nsWinRemoteServer>();
#elif defined(XP_DARWIN)
mRemoteServer = MakeUnique<nsMacRemoteServer>();
#else
return;
#endif

// MISSING ERROR CHECK BEFORE line 172

nsresult rv = mRemoteServer->Startup(mProgram.get(), mProfile.get());

if (NS_FAILED(rv)) {
mRemoteServer = nullptr;
return;
}

nsCOMPtr<nsIObserverService> obs(
do_GetService("@mozilla.org/observer-service;1"));
if (obs) {
obs->AddObserver(this, "xpcom-shutdown", false);
obs->AddObserver(this, "quit-application", false);
}
}

Comment 1

[Michael Hofmann]

The bold lines in the function above were not intended. I just copied the source code - and Bugzilla decided to print some of the ifdef macros in bold typeface.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 3

[Martin Stránský [:stransky] (ni? me)]

Michael, do you mind to create a patch for it?
Thanks.

Comment 4

[Michael Hofmann]

Attachment: patch-firefox-nsRemoteService.cpp

Patch for missing NULL pointer check

Comment 5

[Simona Badau, Desktop QA]

I can't reproduce the crash in Wayland mode on Firefox 84.0.2 nor on the latest Nightly 86.0a1- tried on Ubuntu 20.04 x64 machine since I do not have Gentoo Linux.

Comment 6

[Michael Hofmann]

Simona, if you want to reproduce the crash: compile Firefox with Wayland support AND disable dbus:
--enable-default-toolkit=cairo-gtk3-wayland
--disable-dbus
Start Firefox in Wayland mode:
MOZ_ENABLE_WAYLAND="1" firefox

Comment 7

[Martin Stránský [:stransky] (ni? me)]

Thanks Michael. To land this patch we'd need to go through Phabricator, see https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html for details. Do you mind to take that path?
Thanks.

Comment 8

[Michael Hofmann]

Attachment: Bug 1684365 - Add a null pointer check so that Firefox won't crash when it can't initialize mRemoteServer

Firefox 84.0.1 crashes under Gentoo Linux if it is started in
Wayland mode and if it was compiled WITH Wayland support and
WITHOUT dbus support.

I traced down the problem to line 172 of
toolkit/components/remote/nsRemoteService.cpp:

nsresult rv = mRemoteServer->Startup(mProgram.get(), mProfile.get());

mRemoteServer is NULL and Firefox crashes.

This patch adds a NULL pointer check before that line.

See:

• Mozilla's Bugzilla, bug 1684365
• https://bugs.gentoo.org/762035
• https://forums.gentoo.org/viewtopic-t-1126235.html

Comment 9

[Michael Hofmann]

I added the patch to Phabricator (see previous comment)...

Comment 10

[Michael Hofmann]

Another post about the crash: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11901

Comment 11

[Martin Stránský [:stransky] (ni? me)]

Updated, Thanks.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1edc48b99c80
Add a null pointer check so that Firefox won't crash when it can't initialize mRemoteServer r=stransky

Comment 13

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/1edc48b99c80

Comment 14

[tt_1]

can we get a backport of the fix to firefox beta 86 branch?

Comment 15

[Martin Stránský [:stransky] (ni? me)]

Ehm, do you really run ESR line under Wayland but without DBus? I wonder what's use case for it.