1766901 - Scrape Firefox symbols from Ubuntu snap builds

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, task, P1)

Description

[Gabriele Svelto [:gsvelto]]

As per title, though I should really convince the Ubuntu maintainers to do the uploading on their own.

Comment 1

[Olivier Tilloy]

Gabriele, I'd be happy to help, please let me know what's needed from the snap side.

Comment 2

[Gabriele Svelto [:gsvelto]]

Thank you Olivier! To scrape the symbols I noticed that I can download and unpack the snap package from here, however I don't know where to look for the debug symbols. Alternatively if you could integrate symbol generation and upload into your automated builds I would be very happy and very grateful. I can walk you through what needs to be done to upload symbols for your builds to https://symbols.mozilla.org.

Comment 3

[Olivier Tilloy]

Please note that the snap is currently built with --disable-debug-symbols, so I suppose this wouldn't be usable for your purpose.
We can of course change that to have debug symbols generated at build time. There are however restrictions to what the snap build process (which runs in Launchpad) can do. In particular, it's permitted to access the internet for a limited period of time at the beginning of the build to download dependencies, so uploading symbols towards the end of the build process would likely fail because of this limitation.

If we do build the snap with debug symbols and manage to make them available to the Mozilla Symbol Server, we'll need to keep an eye on the size increase, and potential performance impact?

Comment 4

[Gabriele Svelto [:gsvelto]]

(In reply to Olivier Tilloy from comment #3)

Please note that the snap is currently built with --disable-debug-symbols, so I suppose this wouldn't be usable for your purpose.

Indeed, it's fine to strip the symbols but they need to be there for the Breakpad symbol files to be generated.

We can of course change that to have debug symbols generated at build time. There are however restrictions to what the snap build process (which runs in Launchpad) can do. In particular, it's permitted to access the internet for a limited period of time at the beginning of the build to download dependencies, so uploading symbols towards the end of the build process would likely fail because of this limitation.

Would it be possible to just build the symbols but not upload them? It would be fine if they could be hosted somewhere on your infrastructure, we can make our symbol server regularly download them on its own (see here). Alternatively they could be uploaded by hand at the end of the process even though that's not very practical.

If we do build the snap with debug symbols and manage to make them available to the Mozilla Symbol Server, we'll need to keep an eye on the size increase, and potential performance impact?

Just strip the debug symbols before packaging the build, there's no need to keep them in the snap package once the Breakpad symbol files have been generated. All our release builds are stripped, but we have Breakpad symbol files for them because we generate them during the build.

Comment 5

[Gabriele Svelto [:gsvelto]]

Note: I could dump the stripped executables but the resulting Breakpad symbol files would be of poor quality as only the public symbols would be available (and no filenames/line numbers).

Comment 6

[Sylvestre Ledru [:Sylvestre]]

seems like a P1 S1 bug, no?

Comment 7

[Pascal Chevrel:pascalc]

Agreed that this is an urgent bug to fix now that Ubuntu is shipping Firefox as a snap by default.

Comment 8

[Gabriele Svelto [:gsvelto]]

I just had an idea who might solve the issue w/o causing problems to user. Would it be possible to build another snap package with the exact same configuration as the default one but w/o stripping the debug information? I'd pull it down and extract the symbol files from it, also users who need to debug a problem locally could use that instead of the stripped version. Obviously it would end up under a different package name such as firefox-debug or something like that. WDYT?

Comment 9

[Mike Kaply [:mkaply]]

In particular, it's permitted to access the internet for a limited period of time at the beginning of the build to download dependencies, so uploading symbols towards the end of the build process would likely fail because of this limitation.

Olivier, can you provide more detail on this and if it's a hard limit with no way to solve?

We consider this a pretty serious issue as we're not getting crash details from the Snap.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Any updates on this, Olivier? This is causing a large volume of the Linux crash reports we get to be unusable :(

Comment 11

[Olivier Tilloy]

Sorry, no update on this yet, and I'm off next week, but I'll look into it when I get back.

Comment 12

[Markus Stange [:mstange]]

The lack of symbols was also noticeable in the Firefox Profiler screenshot at the end of this post: https://ubuntu.com/blog/how-are-we-improving-firefox-snap-performance-part-1
In that screenshot, the profiler is showing nonsensical symbols in std::vector. If snap build symbols were present on the Mozilla symbol server, the stacks in the profiler would be correct.

Comment 13

[:gerard-majax]

So out of a few hacking I could do, so far, my understanding is:

• snapcraft wont produce more than one output, i.e., the snap itself
• nightly is built on GitHub Actions, but beta, esr and stable are built on Launchpad
• a snap built using the same set of sources and running under multipass would be expected to repro the same build, and one could see us re-build the same snap as ubuntu on TC and collect its debug symbols ; however, I was told this is not holding for launchpad builds

One could imagine we might be able to bundle the debug symbols within the snap itself, and then rely on unsquashfs to get them, plus snapcraft pack to re-pack without. However, while I can see how we could do that on GitHub Actions, I dont know how the launchpad build service could handle that ; snapcraft remote-build seems to be the tooling, but it's unclear to me how much liberty we have there.

Comment 14

[:gerard-majax]

I tested something: adding a debug-symbols part that depends on firefox part. Looks like in override-pull I can expect to have network, and access to the produced debug symbols:

$ time SNAPCRAFT_BUILD_ENVIRONMENT_MEMORY=96G SNAPCRAFT_BUILD_ENVIRONMENT_CPU=160 snap run snapcraft 2>&1 | tee build.log
Launching a VM.
Skipping pull apikeys (already ran)
Skipping pull rust (already ran)
Skipping pull cbindgen (already ran)
Skipping pull clang (already ran)
Skipping pull distribution (already ran)
Skipping pull ffmpeg (already ran)
Skipping pull firefox-langpacks (already ran)
Skipping pull nodejs (already ran)
Skipping pull pipewire (already ran)
Skipping pull wasi-sdk (already ran)
Skipping pull firefox (already ran)
Cleaning later steps and re-pulling debug-symbols ('override-pull' property changed)
I AM DEBUG
+ echo 'I AM DEBUG'
+ ping -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=6.29 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=8.21 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=6.68 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=115 time=9.32 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 6.286/7.625/9.323/1.215 ms
+ ls -hal ../../firefox/build/obj-x86_64-pc-linux-gnu/dist/firefox-103.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
-rw-r--r-- 1 root root 29M Jun 13 22:40 ../../firefox/build/obj-x86_64-pc-linux-gnu/dist/firefox-103.0a1.en-US.linux-x86_64.crashreporter-symbols.zip

I know it is a hack, but given the state and how badly we need them, could it work on LaunchPad's infra ? I'm not too worried about GitHub Actions for producing Nightly there.

Comment 15

[:gerard-majax]

Assigning since I have a hack PR that does the work and uploads succesfully during a local build on a staging server. We would still have to figure out with Ubuntu:

• Socorro token storage
• If the solution can hit the network during the build

Comment 16

[Olivier Tilloy]

One missing feature on Launchpad (where firefox snap builds happen) is a way to store secrets, to e.g. upload the generated debug symbols to socorro using a private token.

The most straightforward workaround I can think of is to ship the debug symbols somewhere in the snap, and have some server-side logic on https://symbols.mozilla.org/ to fetch and store the symbols for a given snap build. Depending on the size of the symbols, this might be an acceptable solution if it doesn't impact too much the final size of the snap.

Alexandre, do you know offhand how large the symbols are? If not I can experiment to find out myself.

Comment 17

[:gerard-majax]

From my testing, it was generating a < 30MB zip file.

Comment 18

[Gabriele Svelto [:gsvelto]]

(In reply to Olivier Tilloy from comment #16)

The most straightforward workaround I can think of is to ship the debug symbols somewhere in the snap, and have some server-side logic on https://symbols.mozilla.org/ to fetch and store the symbols for a given snap build. Depending on the size of the symbols, this might be an acceptable solution if it doesn't impact too much the final size of the snap.

This is definitely possible! We could have a daily task that checks for new builds, downloads the snaps, extracts the symbols and uploads them to symbols.mozilla.org. This wouldn't be much different compared to what we do now.

Comment 19

[Gabriele Svelto [:gsvelto]]

I've checked the size of the symbols we generate in release builds and they're rather large. I've picked version 97 and the ZIP file is 130 MiB in size. The contents are easily compressible but even zstd and xz slowest compression settings can't get it below at least 60 MiB. Not all symbols are useful though, removing the symbols for tests and harnesses, as well as for the gtest-enabled version of libxul should cut the size down to less than half of that.

Comment 20

[:gerard-majax]

(In reply to Gabriele Svelto [:gsvelto] from comment #19)

I've checked the size of the symbols we generate in release builds and they're rather large. I've picked version 97 and the ZIP file is 130 MiB in size. The contents are easily compressible but even zstd and xz slowest compression settings can't get it below at least 60 MiB. Not all symbols are useful though, removing the symbols for tests and harnesses, as well as for the gtest-enabled version of libxul should cut the size down to less than half of that.

Weird, I had very differents results. I'm re-testing on my side ...

Comment 21

[:gerard-majax]

-rw-r--r-- 1 alexandre alexandre 201M 23 juin  15:58 firefox_103.0a1_amd64_opt+dbgsym.snap
-rw-r--r-- 1 alexandre alexandre 173M 23 juin  20:08 firefox_103.0a1_amd64_opt.snap

There's a 30MB delta between both:

diff --git a/mozconfig b/mozconfig
index 6eaafdb..5064b5b 100644
--- a/mozconfig
+++ b/mozconfig
@@ -4,6 +4,16 @@ ac_add_options --enable-crashreporter
 ac_add_options --with-branding=browser/branding/nightly
 ac_add_options --disable-updater
 ac_add_options --disable-tests
-ac_add_options --enable-debug-symbols
+ac_add_options --disable-debug-symbols
 ac_add_options --disable-bootstrap
 export MOZILLA_OFFICIAL=1
diff --git a/snapcraft.yaml b/snapcraft.yaml
index 7aba15c..456d6f0 100644
--- a/snapcraft.yaml
+++ b/snapcraft.yaml
@@ -205,8 +205,8 @@ parts:
       $MACH buildsymbols
       $MACH install
       DISTRIBUTION=$SNAPCRAFT_PART_INSTALL/usr/lib/firefox/distribution
-      cp obj-*/dist/firefox-*.crashreporter-symbols.zip $DISTRIBUTION/
       mkdir -p $DISTRIBUTION/extensions
+      #cp obj-*/dist/firefox-*.crashreporter-symbols.zip $DISTRIBUTION/
       cp taskcluster/docker/firefox-snap/firefox.desktop $SNAPCRAFT_PART_INSTALL/
       cp browser/branding/nightly/default256.png $SNAPCRAFT_PART_INSTALL/
       sed -i 's/\(^Icon=\).*$/\1\/default256.png/' $SNAPCRAFT_PART_INSTALL/firefox.desktop

Comment 22

[:gerard-majax]

Attachment: WIP: Bug 1766901 - Add TC symbol scrapping

Comment 23

[:gerard-majax]

Current WIP is able to run some things on TC, however I think we need a bit more work to leverage Gabriel's work there:

• [ ] saving the list of packages that were already processed and keeping that on TC's index, like it is already handled for mac symbols
• [ ] ensuring/adding capability in the scripts to ingest this list and only process needed packages
• [ ] add finer granularity of distros to limit the amount of versions we pull (e.g., I see [task 2022-06-24T13:59:03.041Z] package_name = firefox version = 59.0.2+build1-0ubuntu1_amd64 dbg_package_name = firefox)
• [ ] add support for pulling from snap store and extract the symbols there

We should keep in mind we might need more work on Snap side, since some snaps we are linking to (gnome-3-38 ?) might need debug symbols that are not in the apt repos. We might be blocked by that on the lack of infrastructure to handle debug symbols publication by snap.

Comment 24

[:gerard-majax]

I have something able to pull data from Snap Store. However, the Snap Store API has some limitations, including that we can only get a list of teh currently published snap. Since those do not yet contain any debug symbol, it's not yet a big deal but it means we will need to make sure the pull is accurately monitored for success otherwise we could miss data.

If there are several publications of revisions for a snap happening in between our runs, we also risk to miss them.

Comment 25

[Markus Stange [:mstange]]

Note that the size of the .sym files may soon increase by 4x-5x, for bug 1398533. So it's probably worth finding a way to store them outside of the snap.

Comment 26

[:gerard-majax]

(In reply to Markus Stange [:mstange] from comment #25)

Note that the size of the .sym files may soon increase by 4x-5x, for bug 1398533. So it's probably worth finding a way to store them outside of the snap.

We are limited by what Launchpad infrastructure can do so far.

Comment 27

[Phabricator Automation]

Comment on attachment 9282796 [details]
WIP: Bug 1766901 - Add TC symbol scrapping

Revision D150249 was moved to bug 1777423. Setting attachment 9282796 [details] to obsolete.

Comment 28

[:gerard-majax]

Attachment: GitHub Pull Request

Comment 29

[:gerard-majax]

Attachment: GitHub Pull Request (scrap launchpad)

Comment 30

[:gerard-majax]

Attachment: GitHub Pull Request (generate symbols)

Comment 31

[:gerard-majax]

Attachment: Bug 1766901 - Pull symbols for Firefox Snap r?gsvelto!

Comment 32

[:gerard-majax]

Working from Launchpad: https://treeherder.mozilla.org/jobs?repo=try&revision=7d2f2f582f98605a3c3b18cfffcd69b1481f567c

Comment 33

[:gerard-majax]

The current work is able to pull from Launchpad, so we can land this, and as soon as all launchpad-backed build will cherry-pick the required changes, we should get symbols.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8c1587084025
Pull symbols for Firefox Snap r=gsvelto

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last months and this bug has priority 'P1'/severity 'S1'.
:gsvelto, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 36

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/8c1587084025

Comment 37

[Ryan VanderMeulen [:RyanVM]]

Does this need to be uplifted or will the job running on m-c pick up symbols for all supported versions?

Comment 38

[:gerard-majax]

It will pick all versions, there is no need to uplift