Closed Bug 951229 Opened 11 years ago Closed 4 years ago

System symbols not available in Socorro stacks for OS X or Linux

Categories

(Socorro :: Symbols, task)

Product:
Socorro
Component:
Symbols
Platform:
All
macOS
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: smichaud, Unassigned)

References

(Depends on 1 open bug)

Details

Currently no Socorro crash stacks for OS X or Linux contain any non-Mozilla symbols.  Instead, any symbol from a "native" library (one supplied by the OS) is listed as an address -- e.g. CoreGraphics@0x7fb9b (for decode_data on OS X 10.7.X, see bug 925448).

This has been true for a long time, but (at least on OS X) not forever -- e.g. bp-1fbea823-32e7-4482-aaa1-f5b162130605.

I figured someone must be working on this, so I didn't bother to open a bug on it.  But when I looked just now I couldn't find one.  So I opened this bug.
Another example of a Socorro report where "native" symbols were still available (from bug 839752):
bp-213ac5c3-e203-4bbd-a950-3bbfd2130209
Ted, I know we pulled symbols out of OS X installations in the past, do we need to do this again? Do you remember how we did this and/or who did?
Flags: needinfo?(ted)
We don't currently have anything in place to do this. In bug 422527 I did some work to write scripts to scrape symbols off of a machine and upload them to a server, but the machine that was hosting that server went away a while ago and I haven't replaced it. I did get the server component secreviewed in bug 754837, so I don't think there's a huge barrier to getting it set back up, it just needs to happen. I looked into hosting it on paas.allizom.org, but hit some stumbling block and didn't find time to finish it.

Smokey Ardisson was handling this manually for a while in order to improve Camino crash reports but he's no longer doing that:
http://wiki.caminobrowser.org/QA:Crash_Reporting_and_Analysis#Generating_Symbols_for_Mac_OS_X_Libraries

Michael Miller (whom I've hopefully CCed) had a plan for scraping symbols out of Apple's update server, which would be the best way to do this, but it stalled due to lack of resources. It would be great if we could staff that and make it happen. mstange expressed interest in this as well. I can't actually find any record of his proposal, it might have existed solely on IRC.

So in short: the quick path here is to get the webapp back up and running and use the scripts for pulling symbols from local OS X installs. If you'd like to backfill you can hack up one of my scripts and we can push them to the symbol server manually. The longer-term fix is to get a server running with an automated process.
Flags: needinfo?(ted)
Also, we basically gave up on trying to do this for Linux given the huge number of variants of distros etc. There's some discussion in bug 539370.
As Ted said, I wrote a bunch of scripts to unpack the OS X update installers and grab the symbols from all the libraries within. Before I stopped working on it, I had it running in a Debian VM as well so we didn't need an OS X box. The other piece of the puzzle was an update server that would grab the new updates as they appeared from Apple. I was using Reposado (https://github.com/wdas/reposado) to do that.

Let me know if you're interested in those scripts. They may have rotted a little bit but it would at least be a start.
It would be great if you could stick those scripts somewhere that we could poke at them. If this is something we can set up in a bunch of Linux VMs we could almost certainly get it stood up at Mozilla.

Assuming we got something working reliably I'm pretty confident we could make our scraped symbol collection public for others to use.
Here you go! The one thing that's missing is the function dump_breakpad_symbols, but that's basically just the same as the symbol dumping utility you already have. I've included it as well, for completeness sake:

PackageSymbolDumper.py
https://gist.github.com/mrmiller/9655594

DumpBreakpadSymbols.py
https://gist.github.com/mrmiller/9655641

This worked a year ago or so. I haven't tried it more recently, but it should be fairly easy to adapt if the format's changed slightly.
mstange was looking into this but got hung up because he was trying to run the scripts on Linux and didn't have a dump_syms binary that worked. As part of my work on bug 543111 I fixed that:
https://people.mozilla.com/~tmielczarek/mac_dump_syms_for_linux

(I'll have the patches up for review eventually so that you could build this out-of-the-box.)
Summary: "Native" symbols not available in Socorro stacks for OS X or Linux → System symbols not available in Socorro stacks for OS X or Linux
FYI, building the latest Breakpad from source with `./configure && make` will now build the src/tools/mac/dump_syms/dump_syms binary that can be used to dump symbols from Mach-O binaries.

n.b. the Breakpad source recently moved to git:
https://chromium.googlesource.com/breakpad/breakpad/+/master/README#16
(In reply to michaelrmmiller+bugzilla from comment #7)
> Here you go! The one thing that's missing is the function
> dump_breakpad_symbols, but that's basically just the same as the symbol
> dumping utility you already have. I've included it as well, for completeness
> sake:

I did a bunch of hacking on this recently:
https://github.com/luser/breakpad-mac-update-symbols

This is working for me locally, I'm going to try to get it running in our Taskcluster CI environment.
I ran this in Taskcluster:
https://tools.taskcluster.net/task-graph-inspector/#XhNHyG__SryUwmHpD1SCJA

It succeeded (although the upload symbols bit claims to have failed).
About 17 of the "50 topcrashes" for Aurora on Mac are in system libraries. That includes functions we'd want to skiplist in order to break down by caller, like abort() in bug 1267288.
With bug 1264367 fixed we will start getting full Build IDs for modules in Linux crash reports, so it'd be more straightforward to fill system symbols from distro packages.
Depends on: 539370
I finally got bug 1301471 working today, so this should be fixed for OS X system libraries for the foreseeable future. The only thing left to do here is Linux symbols in bug 539370.
Bumping this over to the symbols component.
Component: General → Symbols

Gabriele: Is this a duplicate of other bugs you and others are working on for getting system symbols into Symbols?

Flags: needinfo?(gsvelto)

Yeah, we've got everything covered now so I think we can close it.

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(gsvelto)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.