1859267 - Figure out why crashy glib messages don't trigger the crash reporter.

Status: RESOLVED FIXED

(Core :: Widget: Gtk, defect)

Description

[Emilio Cobos Álvarez [:emilio]]

As reported in bug 1752398 comment 17.

Comment 1

[Martin Stránský [:stransky] (ni? me)]

It's because we're terminated from Gtk library by _exit() call.
See https://github.com/GNOME/gtk/blob/main/gdk/wayland/gdkeventsource.c for instance.
If wayland display socked is closed by compositor and/or the connection is broken Gtk just terminates the application with _exit() call.

Comment 2

[Martin Stránský [:stransky] (ni? me)]

I don't think this is actionable from our side.

Comment 3

[Martin Stránský [:stransky] (ni? me)]

Note that this is true "lost connection" so we're just unplugged and potential backtrace will contain a stack with gtk event loop and nothing else. It doesn't add much value to us. We already handle wl_log() messages which carries at least some useful info why we're disconnected.

Comment 4

[The 8472]

Imo the issue is that these go unreported, which means - afaik - there's no way to know how often it crashes under wayland due to these errors.

Comment 5

[Martin Stránský [:stransky] (ni? me)]

(In reply to The 8472 from comment #4)

Imo the issue is that these go unreported, which means - afaik - there's no way to know how often it crashes under wayland due to these errors.

Yes, you're right. But such termination usually means fatal state of Wayland compositor itself. I'd expect it on Sway but not on Gnome/Mutter.

Comment 6

[Emilio Cobos Álvarez [:emilio]]

This is because g_message with modern glib versions uses a different API we can hook.

Comment 7

[Emilio Cobos Álvarez [:emilio]]

On Sway this is easy to reproduce by busy looping in the browser toolbox. I used sway to confirm that https://phabricator.services.mozilla.com/D191074 works.

Comment 8

[Emilio Cobos Álvarez [:emilio]]

Attachment: Bug 1859267 - Hook glib structured logging too. r=stransky

Comment 9

[Laurențiu Nicola]

Anecdotally, after using GNOME Wayland since 2016 or so, Mutter is more like to crash than Firefox.

Comment 10

[Emilio Cobos Álvarez [:emilio]]

Yeah, Mutter and KWin have workarounds to prevent this kind of crash...

Comment 11

[The 8472]

See bug 1792754 for another way to produce crashes under sway.

Comment 12

[Martin Stránský [:stransky] (ni? me)]

I hesitant to collect such crash reports. Do we really want to track Wayland compositor crashes? I don't think we want to care or fix Sway crashes but that will pollute our crash pool with non-firefox crashes.

Emilio, can we separate compositor and Firefox crashes if your patch lands?

Comment 13

[Emilio Cobos Álvarez [:emilio]]

If the compositor crashes it's unlikely the crash reporter window will show up, and that would prevent at least some of the crashes going through, right?

If we still get non-actionable crashes, then we can try to differentiate those, maybe by trying to connect to the wayland compositor again and not reporting them if so. But it's not too different than receiving a SIGTERM when the computer powers off, and in that case we also try to collect a crash report I believe.

Gabriele might know more.

Comment 14

[The 8472]

They're not compositor crashes. The compositor keeps running fine.
They're the compositor severing the unix socket because the client application does not keep up with event processing (which leads to the socket buffer being full) and then the client library (glib) interpreting this disconnect as a fatal problem instead of reconnecting and recovering state (as Qt supposedly does).

Comment 15

[Emilio Cobos Álvarez [:emilio]]

(In reply to The 8472 from comment #14)

They're not compositor crashes. The compositor keeps running fine.

Sometimes, but for actual compositor crashes, the crash on our end looks the same. I think that's what Martin is concerned about.

Comment 16

[Gabriele Svelto [:gsvelto]]

We don't catch SIGTERM because AFAIR we handle it and perform an orderly shutdown. In this case, if you want to tell these crashes apart from others, you could add some easy-to-filter prefix to the crash reason. Something like: "GTK error: <actual_error_goes_here>". This way we could write a rule on Socorro that tells this class of crashes apart from the others. This could also be a crash annotation, your choice.

Comment 17

[Emilio Cobos Álvarez [:emilio]]

The current patch should be easy in the sense that we'd crash in the log function so that should be the last frame on the stack. I think martin was more concerned about distinguishing some of these crashes from others which might not be Firefox bugs.

Martin, I still think the current version of the patch is useful, if only to know which desktop environments we crash on more etc (for example, if 99% of these crashes are on sway, we might want not to enable wayland there yet or something).

Comment 18

[Martin Stránský [:stransky] (ni? me)]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #17)

The current patch should be easy in the sense that we'd crash in the log function so that should be the last frame on the stack. I think martin was more concerned about distinguishing some of these crashes from others which might not be Firefox bugs.

Martin, I still think the current version of the patch is useful, if only to know which desktop environments we crash on more etc (for example, if 99% of these crashes are on sway, we might want not to enable wayland there yet or something).

Okay, let's try that and we can adjust it later if we have more info.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9b7c81791d7f
Hook glib structured logging too. r=stransky

Comment 20

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/9b7c81791d7f

Comment 21

[The 8472]

I have tested this on sway and can confirm that crash reports are getting submitted now.

Though it's still too crashy for me and I'll keep using it under X11 instead, so I won't be submitting reports regularly.