Closed Bug 1766901 Opened 2 years ago Closed 2 years ago

Scrape Firefox symbols from Ubuntu snap builds

Categories

(Toolkit :: Crash Reporting, task, P1)

Product:
Toolkit
Component:
Crash Reporting
Type:
task

Tracking

()

Status:
RESOLVED FIXED
Milestone:
105 Branch
Tracking Flags:
Tracking Status
firefox-esr91 --- fixed
firefox-esr102 --- fixed
firefox100 + wontfix
firefox101 + wontfix
firefox102 + wontfix
firefox103 --- fixed
firefox104 --- fixed
firefox105 --- fixed

People

(Reporter: gsvelto, Assigned: gerard-majax)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(4 files, 1 obsolete file)

GitHub Pull Request
49 bytes, text/x-github-pull-request
Details | Review
GitHub Pull Request (scrap launchpad)
49 bytes, text/x-github-pull-request
Details | Review
GitHub Pull Request (generate symbols)
48 bytes, text/x-github-pull-request
Details | Review
Bug 1766901 - Pull symbols for Firefox Snap r?gsvelto!
48 bytes, text/x-phabricator-request
Details | Review

As per title, though I should really convince the Ubuntu maintainers to do the uploading on their own.

Blocks: snap

Gabriele, I'd be happy to help, please let me know what's needed from the snap side.

Flags: needinfo?(gsvelto)

Thank you Olivier! To scrape the symbols I noticed that I can download and unpack the snap package from here, however I don't know where to look for the debug symbols. Alternatively if you could integrate symbol generation and upload into your automated builds I would be very happy and very grateful. I can walk you through what needs to be done to upload symbols for your builds to https://symbols.mozilla.org.

Flags: needinfo?(gsvelto)

Please note that the snap is currently built with --disable-debug-symbols, so I suppose this wouldn't be usable for your purpose.
We can of course change that to have debug symbols generated at build time. There are however restrictions to what the snap build process (which runs in Launchpad) can do. In particular, it's permitted to access the internet for a limited period of time at the beginning of the build to download dependencies, so uploading symbols towards the end of the build process would likely fail because of this limitation.

If we do build the snap with debug symbols and manage to make them available to the Mozilla Symbol Server, we'll need to keep an eye on the size increase, and potential performance impact?

(In reply to Olivier Tilloy from comment #3)

Please note that the snap is currently built with --disable-debug-symbols, so I suppose this wouldn't be usable for your purpose.

Indeed, it's fine to strip the symbols but they need to be there for the Breakpad symbol files to be generated.

We can of course change that to have debug symbols generated at build time. There are however restrictions to what the snap build process (which runs in Launchpad) can do. In particular, it's permitted to access the internet for a limited period of time at the beginning of the build to download dependencies, so uploading symbols towards the end of the build process would likely fail because of this limitation.

Would it be possible to just build the symbols but not upload them? It would be fine if they could be hosted somewhere on your infrastructure, we can make our symbol server regularly download them on its own (see here). Alternatively they could be uploaded by hand at the end of the process even though that's not very practical.

If we do build the snap with debug symbols and manage to make them available to the Mozilla Symbol Server, we'll need to keep an eye on the size increase, and potential performance impact?

Just strip the debug symbols before packaging the build, there's no need to keep them in the snap package once the Breakpad symbol files have been generated. All our release builds are stripped, but we have Breakpad symbol files for them because we generate them during the build.

Note: I could dump the stripped executables but the resulting Breakpad symbol files would be of poor quality as only the public symbols would be available (and no filenames/line numbers).

seems like a P1 S1 bug, no?

Agreed that this is an urgent bug to fix now that Ubuntu is shipping Firefox as a snap by default.

Severity: -- → S1
status-firefox100: --- → affected
status-firefox101: --- → affected
status-firefox102: --- → affected
tracking-firefox100: --- → ?
tracking-firefox101: --- → ?
Priority: -- → P1

I just had an idea who might solve the issue w/o causing problems to user. Would it be possible to build another snap package with the exact same configuration as the default one but w/o stripping the debug information? I'd pull it down and extract the symbol files from it, also users who need to debug a problem locally could use that instead of the stripped version. Obviously it would end up under a different package name such as firefox-debug or something like that. WDYT?

Flags: needinfo?(olivier)

In particular, it's permitted to access the internet for a limited period of time at the beginning of the build to download dependencies, so uploading symbols towards the end of the build process would likely fail because of this limitation.

Olivier, can you provide more detail on this and if it's a hard limit with no way to solve?

We consider this a pretty serious issue as we're not getting crash details from the Snap.

Any updates on this, Olivier? This is causing a large volume of the Linux crash reports we get to be unusable :(

status-firefox100: affected → wontfix
status-firefox-esr91: --- → affected
tracking-firefox-esr91: --- → ?

Sorry, no update on this yet, and I'm off next week, but I'll look into it when I get back.

The lack of symbols was also noticeable in the Firefox Profiler screenshot at the end of this post: https://ubuntu.com/blog/how-are-we-improving-firefox-snap-performance-part-1
In that screenshot, the profiler is showing nonsensical symbols in std::vector. If snap build symbols were present on the Mozilla symbol server, the stacks in the profiler would be correct.

So out of a few hacking I could do, so far, my understanding is:

  • snapcraft wont produce more than one output, i.e., the snap itself
  • nightly is built on GitHub Actions, but beta, esr and stable are built on Launchpad
  • a snap built using the same set of sources and running under multipass would be expected to repro the same build, and one could see us re-build the same snap as ubuntu on TC and collect its debug symbols ; however, I was told this is not holding for launchpad builds

One could imagine we might be able to bundle the debug symbols within the snap itself, and then rely on unsquashfs to get them, plus snapcraft pack to re-pack without. However, while I can see how we could do that on GitHub Actions, I dont know how the launchpad build service could handle that ; snapcraft remote-build seems to be the tooling, but it's unclear to me how much liberty we have there.

I tested something: adding a debug-symbols part that depends on firefox part. Looks like in override-pull I can expect to have network, and access to the produced debug symbols:

$ time SNAPCRAFT_BUILD_ENVIRONMENT_MEMORY=96G SNAPCRAFT_BUILD_ENVIRONMENT_CPU=160 snap run snapcraft 2>&1 | tee build.log
Launching a VM.
Skipping pull apikeys (already ran)
Skipping pull rust (already ran)
Skipping pull cbindgen (already ran)
Skipping pull clang (already ran)
Skipping pull distribution (already ran)
Skipping pull ffmpeg (already ran)
Skipping pull firefox-langpacks (already ran)
Skipping pull nodejs (already ran)
Skipping pull pipewire (already ran)
Skipping pull wasi-sdk (already ran)
Skipping pull firefox (already ran)
Cleaning later steps and re-pulling debug-symbols ('override-pull' property changed)
I AM DEBUG
+ echo 'I AM DEBUG'
+ ping -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=6.29 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=8.21 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=6.68 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=115 time=9.32 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 6.286/7.625/9.323/1.215 ms
+ ls -hal ../../firefox/build/obj-x86_64-pc-linux-gnu/dist/firefox-103.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
-rw-r--r-- 1 root root 29M Jun 13 22:40 ../../firefox/build/obj-x86_64-pc-linux-gnu/dist/firefox-103.0a1.en-US.linux-x86_64.crashreporter-symbols.zip

I know it is a hack, but given the state and how badly we need them, could it work on LaunchPad's infra ? I'm not too worried about GitHub Actions for producing Nightly there.

Assigning since I have a hack PR that does the work and uploads succesfully during a local build on a staging server. We would still have to figure out with Ubuntu:

  • Socorro token storage
  • If the solution can hit the network during the build
Assignee: nobody → lissyx+mozillians

One missing feature on Launchpad (where firefox snap builds happen) is a way to store secrets, to e.g. upload the generated debug symbols to socorro using a private token.

The most straightforward workaround I can think of is to ship the debug symbols somewhere in the snap, and have some server-side logic on https://symbols.mozilla.org/ to fetch and store the symbols for a given snap build. Depending on the size of the symbols, this might be an acceptable solution if it doesn't impact too much the final size of the snap.

Alexandre, do you know offhand how large the symbols are? If not I can experiment to find out myself.

Flags: needinfo?(olivier) → needinfo?(lissyx+mozillians)

From my testing, it was generating a < 30MB zip file.

Flags: needinfo?(lissyx+mozillians)

(In reply to Olivier Tilloy from comment #16)

The most straightforward workaround I can think of is to ship the debug symbols somewhere in the snap, and have some server-side logic on https://symbols.mozilla.org/ to fetch and store the symbols for a given snap build. Depending on the size of the symbols, this might be an acceptable solution if it doesn't impact too much the final size of the snap.

This is definitely possible! We could have a daily task that checks for new builds, downloads the snaps, extracts the symbols and uploads them to symbols.mozilla.org. This wouldn't be much different compared to what we do now.

I've checked the size of the symbols we generate in release builds and they're rather large. I've picked version 97 and the ZIP file is 130 MiB in size. The contents are easily compressible but even zstd and xz slowest compression settings can't get it below at least 60 MiB. Not all symbols are useful though, removing the symbols for tests and harnesses, as well as for the gtest-enabled version of libxul should cut the size down to less than half of that.

(In reply to Gabriele Svelto [:gsvelto] from comment #19)

I've checked the size of the symbols we generate in release builds and they're rather large. I've picked version 97 and the ZIP file is 130 MiB in size. The contents are easily compressible but even zstd and xz slowest compression settings can't get it below at least 60 MiB. Not all symbols are useful though, removing the symbols for tests and harnesses, as well as for the gtest-enabled version of libxul should cut the size down to less than half of that.

Weird, I had very differents results. I'm re-testing on my side ...

-rw-r--r-- 1 alexandre alexandre 201M 23 juin  15:58 firefox_103.0a1_amd64_opt+dbgsym.snap
-rw-r--r-- 1 alexandre alexandre 173M 23 juin  20:08 firefox_103.0a1_amd64_opt.snap

There's a 30MB delta between both:

diff --git a/mozconfig b/mozconfig
index 6eaafdb..5064b5b 100644
--- a/mozconfig
+++ b/mozconfig
@@ -4,6 +4,16 @@ ac_add_options --enable-crashreporter
 ac_add_options --with-branding=browser/branding/nightly
 ac_add_options --disable-updater
 ac_add_options --disable-tests
-ac_add_options --enable-debug-symbols
+ac_add_options --disable-debug-symbols
 ac_add_options --disable-bootstrap
 export MOZILLA_OFFICIAL=1
diff --git a/snapcraft.yaml b/snapcraft.yaml
index 7aba15c..456d6f0 100644
--- a/snapcraft.yaml
+++ b/snapcraft.yaml
@@ -205,8 +205,8 @@ parts:
       $MACH buildsymbols
       $MACH install
       DISTRIBUTION=$SNAPCRAFT_PART_INSTALL/usr/lib/firefox/distribution
-      cp obj-*/dist/firefox-*.crashreporter-symbols.zip $DISTRIBUTION/
       mkdir -p $DISTRIBUTION/extensions
+      #cp obj-*/dist/firefox-*.crashreporter-symbols.zip $DISTRIBUTION/
       cp taskcluster/docker/firefox-snap/firefox.desktop $SNAPCRAFT_PART_INSTALL/
       cp browser/branding/nightly/default256.png $SNAPCRAFT_PART_INSTALL/
       sed -i 's/\(^Icon=\).*$/\1\/default256.png/' $SNAPCRAFT_PART_INSTALL/firefox.desktop

Current WIP is able to run some things on TC, however I think we need a bit more work to leverage Gabriel's work there:

  • [ ] saving the list of packages that were already processed and keeping that on TC's index, like it is already handled for mac symbols
  • [ ] ensuring/adding capability in the scripts to ingest this list and only process needed packages
  • [ ] add finer granularity of distros to limit the amount of versions we pull (e.g., I see [task 2022-06-24T13:59:03.041Z] package_name = firefox version = 59.0.2+build1-0ubuntu1_amd64 dbg_package_name = firefox)
  • [ ] add support for pulling from snap store and extract the symbols there

We should keep in mind we might need more work on Snap side, since some snaps we are linking to (gnome-3-38 ?) might need debug symbols that are not in the apt repos. We might be blocked by that on the lack of infrastructure to handle debug symbols publication by snap.

I have something able to pull data from Snap Store. However, the Snap Store API has some limitations, including that we can only get a list of teh currently published snap. Since those do not yet contain any debug symbol, it's not yet a big deal but it means we will need to make sure the pull is accurately monitored for success otherwise we could miss data.

If there are several publications of revisions for a snap happening in between our runs, we also risk to miss them.

Note that the size of the .sym files may soon increase by 4x-5x, for bug 1398533. So it's probably worth finding a way to store them outside of the snap.

(In reply to Markus Stange [:mstange] from comment #25)

Note that the size of the .sym files may soon increase by 4x-5x, for bug 1398533. So it's probably worth finding a way to store them outside of the snap.

We are limited by what Launchpad infrastructure can do so far.

Depends on: 1777423

Comment on attachment 9282796 [details]
WIP: Bug 1766901 - Add TC symbol scrapping

Revision D150249 was moved to bug 1777423. Setting attachment 9282796 [details] to obsolete.

Attachment #9282796 - Attachment is obsolete: true
Summary: Scrape symbols from Ubuntu snap builds → Scrape Firefox symbols from Ubuntu snap builds
Blocks: 1778359
Attached file GitHub Pull Request —
Blocks: 1781041

The current work is able to pull from Launchpad, so we can land this, and as soon as all launchpad-backed build will cherry-pick the required changes, we should get symbols.

Pushed by alissy@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8c1587084025
Pull symbols for Firefox Snap r=gsvelto

The bug assignee didn't login in Bugzilla in the last months and this bug has priority 'P1'/severity 'S1'.
:gsvelto, could you have a look please?
For more information, please visit auto_nag documentation.

Assignee: lissyx+mozillians → nobody
Flags: needinfo?(gsvelto)
Assignee: nobody → lissyx+mozillians
Flags: needinfo?(gsvelto)

https://hg.mozilla.org/mozilla-central/rev/8c1587084025

Status: NEW → RESOLVED
Closed: 2 years ago
status-firefox105: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 105 Branch

Does this need to be uplifted or will the job running on m-c pick up symbols for all supported versions?

status-firefox102: affected → wontfix
status-firefox103: --- → fix-optional
status-firefox104: --- → affected
status-firefox-esr102: --- → affected
Flags: needinfo?(lissyx+mozillians)

It will pick all versions, there is no need to uplift

Flags: needinfo?(lissyx+mozillians)
Blocks: 1782734
Depends on: 1786470
Depends on: 1786511
Depends on: 1840795
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: