1709543 - crash symbols for recent macOS beta versions

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

@ libobjc.A.dylib@0x6829 and @ libobjc.A.dylib@0x681d are unsymbolified.

Platform versions:
11.3.0 20E5217a
11.4.0 20F5055c

Comment 1

[Gabriele Svelto [:gsvelto]]

Steven sent me a batch of new symbols yesterday, I'll upload them shortly.

Comment 2

[Steven Michaud [:smichaud] (Retired)]

The symbols I sent to Gabriele won't help in this case. They're for macOS 11.3.1 (build 20E241), which was released by Apple two days ago. Both of the builds from comment #0 are beta builds. (See https://en.wikipedia.org/wiki/MacOS_Big_Sur#Release_history.)

As far as I know, Mozilla has only ever done symbol scraping for release versions of macOS -- not beta versions. Mozilla does have a system for automated scraping of macOS symbols. But it's never worked very well. And as far as I know it doesn't work at all for macOS 11 (Big Sur). So additional manual symbol scraping has always been required. For the last couple of years I've been the one doing it.

Adding manual scraping for beta builds would double or triple my work. I'm not willing to put in that kind of time and effort -- at least not on a regular basis. I did scrape symbols for the last few pre-release betas of macOS Big Sur, but only because lots of Firefox users were using it (and crashing on it) in the two or three months leading up to its initial release.

I was a Mozilla employee for many years until I retired in 2015. Then I spent a few years completely uninvolved with Mozilla. But recently (in 2019) I started working again on Mozilla bugs as a contributor. I've concentrated on bugs that are basically Apple bugs, which require reverse engineering of macOS to resolve, and improving the quality of macOS crash reports at https://crash-stats.mozilla.org/ (by patching Breakpad and doing my manual symbol scraping). For testing my own GitHub projects I've created installations (VMWare VMs) of all major releases of macOS or OS X going back to 10.9.5. So it's been easy for me to add the infrastructure needed for manual symbol scraping, and to run it every time Apple does a new major or minor release. But I draw the line at beta releases.

Like I said, Mozilla has (or had) a system for automatically scraping symbols on macOS 10.15 (Catalina) and 10.14 (Mojave). But it doesn't work on macOS 11, and presumably won't work on any future major releases of macOS. As I understand it, Mozilla is now working on a new system for keeping track of symbols, which would no longer require any kind of symbol scraping (automatic or manual). This would use services provided by a company called Sentry. But it won't be finished for quite a while. And during that time manual scraping will continue to be required. Gabriele knows a lot more about this than I do. He may wish to comment further.

A couple of months ago I made a proposal to a couple of Mozilla employees, including Gabriele, that Mozilla buy a fairly beefy Mac Intel box, on which to run virtual machines (from VMware or Parallels) for each version of macOS that requires symbols scraping. (Apple usually maintains support (provides security updates) for the current version of macOS and the two previous versions.) I did this after I discovered that Apple now makes available installers for older versions of macOS. My proposal got a positive reaction. But as far as I know, nothing more has been done in the time since.

Once Mozilla is doing the manual scraping, they can choose to do it on beta releases of macOS, if that seems appropriate.

Comment 3

[Gabriele Svelto [:gsvelto]]

Ouch, if those are beta versions then I'm afraid this will be a WONTFIX. We've discussed Steven's proposal and fleshed out how to implement using the existing infra but there's no one with spare cycles available to do it at the moment. I'm trying to figure out if Sentry is also doing this scraping. We're discussing syncing our symbol servers so that each gets the symbols the other is missing but I think it will still take us a few months to do it.

Comment 4

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1709543 - Fix the macOS symbol scraper to scrape older builds r=glandium

This patch fixes several issues and needs to work together with matching
changes in reposado:

• The package dumper handles nested packages correctly
• Disk image files (.dmg) are handled as well as package ones (.pkg)
• Fixed some code that still used python 2 semantics
• Deleting temporary directories was made more reliable
• The PBZX file parser had several bugs that prevented it from working with
  all but the simplest archives. This was fixed and it was converted to
  python 3.

Comment 5

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1709543 - Stop downloading packages when we have more than 10 GiB of them r=glandium

Depends on D137207

Comment 6

[Markus Stange [:mstange]]

When I tried to get the libraries for macOS 12.3.1 manually, I also had to decrypt a ramdisk dmg along the way, using asn1rdskdecode.py, as described here under "Getting the Files": https://blogs.blackberry.com/en/2021/05/strong-arming-with-macos-adventures-in-cross-platform-emulation

Comment 7

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1709543 - Only list macOS installation packages up to version 10.15 r=glandium

We need to modify the dmg tool to extract newer disk images and we lack
support for unpacking YAA archives which are used starting with macOS 11.
Once those issues are addressed we can revert this.

Depends on D151176

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6c0cc35a9d58
Fix the macOS symbol scraper to scrape older builds r=glandium
https://hg.mozilla.org/integration/autoland/rev/8f7328675c3f
Stop downloading packages when we have more than 10 GiB of them r=glandium
https://hg.mozilla.org/integration/autoland/rev/4c6e650fe33d
Only list macOS installation packages up to version 10.15 r=glandium

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/6c0cc35a9d58
https://hg.mozilla.org/mozilla-central/rev/8f7328675c3f
https://hg.mozilla.org/mozilla-central/rev/4c6e650fe33d