422527 - figure out a way to get up-to-date OS X symbols for system libraries

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[(not currently active) Ted Mielczarek]

I uploaded some symbols for OS X 10.4.9 (or whatever was on my computer at the time), but obviously the world has moved on since then. Bug 419882 is about using the symbol server to grab recent symbols for Windows, but I'd like to try to stay up to date on Mac symbols too. I'm thinking maybe I can just run a script semi-regularly on my Mac (10.5.2 currently) to extract symbols from system libs, and then rsync it to the os_symbols dir. Ideally we'd do the same thing on a 10.4 machine, but I totally don't have an extra mac.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Ted, is this still being done ad-hoc?  (Is that what we did for Talkback, too?)

Comment 2

[(not currently active) Ted Mielczarek]

Yes. I haven't even done it in a while because it's a pain. If you can figure out something better, I'm all ears. I started on a script to handle Win32 via the Microsoft Symbol Server over in bug 419882 but that didn't get finished.

I don't know what was done for Talkback, I was never involved with that. From what I've heard, OS symbols were just not available.

Comment 3

[(not currently active) Ted Mielczarek]

Alternately, if you'd just like to take ownership of uploading ad-hoc system symbols for OS X, we can get you the right permissions to do that.

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

In talking with Stuart this week, it's become clear to us that 1) Camino is really going to need OS symbols for Breakpad to be useful to us, and 2) even if I'm not able to get OS symbols up until a week after a new minor OS version and can't provide any backfilled versions, that's still better than the status quo.  So let me know what sort of pain I have just inflicted upon myself ;)

Comment 5

[(not currently active) Ted Mielczarek]

File an IT bug, you'll need access to dm-symbolpush01 and write access to /mnt/netapp/breakpad/symbols_os . I've just been uploading them ad-hoc there. It basically just boils down to running symbolstore.py, passing it all the system libraries you want symbols for on the command line.  I think I used to have a script for that, but I can't locate it at the moment...

Comment 6

[Laura Thomson :laura]

Is this still an issue?

Comment 7

[(not currently active) Ted Mielczarek]

Yes, although it's not a Socorro webapp issue. I guess it would be nice to have a place to stick things like this that aren't Breakpad issues, but also aren't Socorro issues, but still affect crash reporting.

Comment 8

[Jeff Muizelaar [:jrmuizel]]

This would be nice for crashes like:
https://crash-stats.mozilla.com/report/index/d9bf9f3a-1e10-4aa7-afbc-ed81c2110208

Comment 9

[(not currently active) Ted Mielczarek]

I landed Mark Mentovai's patch to fix dumping symbols from OS X system libraries upstream:
http://code.google.com/p/google-breakpad/source/detail?r=778

I've also been working on a few other things that I hope to use to fix this bug:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/rest-symbol-server/
http://hg.mozilla.org/users/tmielczarek_mozilla.com/mac-breakpad-symbol-gather/

My plan is to set the symbol server bit up somewhere (probably brasstacks), and then crowdsource the actual symbol gathering by having people run the gathersymbols script on their Mac systems as a cron job. I can then upload the data from the server to our symbol store and hopefully collect a reasonable set of OS X system symbols.

Comment 10

[(not currently active) Ted Mielczarek]

I uploaded symbols for recent 10.5 and 10.6 to the symbol server using this code, so we ought to see some more useful symbols in reports starting now. I'm going to get the server bit setup on brasstacks and then we can get the scripts running on a regular basis.

Comment 11

[(not currently active) Ted Mielczarek]

I've got this up and running on brasstacks.mozilla.org. I've got a 10.5 and 10.6 machine handy, so I've run it on both of those, but if someone has a 10.4 machine and wants to run it there as well, we could get some more symbols. Additionally, it might be worthwhile to find machines without all the security fixes installed and run it to get older symbol versions, but I'm not going to do that work. The script + binary can be found here:
http://people.mozilla.com/~tmielczarek/gathersymbols.zip

You can just run gathersymbols.py from there and it will upload to brasstacks. I have a script on brasstacks that takes new symbols and uploads them to the symbol server.