1910890 - All Ubuntu 24.04 LTS crashes are missing symbols

Status: RESOLVED FIXED

(Thunderbird :: Build Config, task)

Description

[Wayne Mery (:wsmwk)]

We are not getting symbols for some subset of Ubuntu crashes.

For Ubuntu * (including 24.04 LTS, Ubuntu 22.04.4 LTS, Ubuntu 23.10)

• Almost all esr 128 crashes of the last 6 months are lacking symbols

• Indeed we don't have symbols for any Thunderbird versions

But some of the crashes do have symbols as seen in this crash query

Comment 1

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #0)

We are not getting symbols for some subset of Ubuntu crashes.

For Ubuntu * (including 24.04 LTS, Ubuntu 22.04.4 LTS, Ubuntu 23.10)

....
But some of the crashes do have symbols as seen in this crash query

Turns out all of those are pre-128 crash reports, so of not much interest anymore.

But of version 128 crashes, for the #1 linux crash signature libxul.so | libnspr4.so, 85% are Ubuntu 24.x and have no symbols. In other words, we have little visibility into the crash behavior of our largest linux user population. This might be of greater concern going into monthly release cadence.

But if we could get the symbols ... Firefox seems to for a majority of their crashes.

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #0)

We are not getting symbols for some subset of Ubuntu crashes.

For Ubuntu * (including 24.04 LTS, Ubuntu 22.04.4 LTS, Ubuntu 23.10)

• Almost all esr 128 crashes of the last 6 months are lacking symbols

Updated list of crash sigs

Comment 3

[Corey Bryant]

I suspect this is snap related if it's limited to Ubuntu.
@Daniel, do you have any thoughts on this?

Comment 4

[Daniel Darnell [:dandarnell]]

Yes, it does look likely that this is snap related, but I'm not 100% sure. Let me do a little more digging, and I'll file a bug on the Thunderbird snap.

Comment 5

[Heather Ellsworth]

I believe we can add some debug symbols into the thunderbird snap similar to what they do in the firefox snap

Comment 6

[Daniel Darnell [:dandarnell]]

Attachment: Bug 1910890 - Enable repackaging Thunderbird w/ debug symbols as Snap. r=#thunderbuild

Comment 7

[seb128]

indeed, snaps don't have debug symbols out of the box. For the Firefox snap we generate a debug artifact during the (https://github.com/canonical/firefox-snap/blob/a6e1c6bf7c022fa817b712f48db2d11128ccd0e4/snapcraft.yaml#L589) and Mozilla has a collector job on their side which watch the snap builds on launchpad and import the symbols to their service. Note that you would also want to import the gnome-sdk symbols

Comment 8

[Heather Ellsworth]

Adding a debug-symbols part like in the firefox snap should be easy enough.. Daniel, from the infrastructure/build side, could you look into the mozilla collector job and see how we could implement the same thing?

Comment 9

[Daniel Darnell [:dandarnell]]

I think in our case (at least for our Taskcluster snap) we'd simply repack with the debug build of Thunderbird rather than the optimized build. That's what my WIP patch does. The patch should be ready, but I want to wait until Rob is available to get a second pair of eyes on it.

I'm looking into doing it FF's way, but the main reason we haven't yet has been due to the amount of build time it would add to releases. However, it may be the way to go moving forward.

Comment 10

[Corey Bryant]

It appears the Debian package and the Ubuntu PPA also don't provide debug symbols.

Based on bug 1943114, it appears that Debian bookworm doesn't provide debug symbols for Thunderbird. I've opened a bug for the Debian package to enable debug symbols.

And based on bug 1906482, it appears that https://launchpad.net/~mozillateam/+archive/ubuntu/ppa is also missing debug symbols. I've reached out to the Ubuntu PPA maintainers to see if they can enable debug symbols.

Comment 11

[Wayne Mery (:wsmwk)]

see bug 1766901 for what firefox did with ubuntu

Comment 12

[Gabriele Svelto [:gsvelto]]

I've checked Ubuntu PPA's and they do generate debug information for Thunderbird builds (see here) but our scripts are not scraping them like we scrape Firefox PPA builds. I'll open a bug to add them to the scrapers.

Comment 13

[Heather Ellsworth]

Attachment: GitHub Pull Request Beta

Comment 14

[Heather Ellsworth]

Attachment: GitHub Pull Request Stable

Comment 15

[Heather Ellsworth]

Attachment: GitHub Pull Request Stable Core24

Comment 16

[Heather Ellsworth]

:gerard-majax made the required changes for this. Thank you so much!

Comment 17

[Gabriele Svelto [:gsvelto]]

Whenever you have issues with crashes reach out to the #crashreporting channel on Matrix. We've got your back.

Comment 18

[Corey Bryant]

(In reply to Corey Bryant from comment #10)

Based on bug 1943114, it appears that Debian bookworm doesn't provide debug symbols for Thunderbird. I've opened a bug for the Debian package to enable debug symbols.

Correction: The Debian package does provide debug symbols. See update in bug for how to enable.

Comment 19

[Daniel Darnell [:dandarnell]]

Although I am temporarily disabling snap repackaging in Taskcluster (bug 1951789, this is separate from Launchpad/Canonical's build), I would still like to land my debug symbols patch. It will be useful when Taskcluster snap repackaging is re-enabled in the future.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1fe583e7f353
Enable repackaging Thunderbird w/ debug symbols as Snap. r=coreyb

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5e3e75b86737
follow-up, fix linting. rs=lint DONTBUILD