492238 - Script to facilitate generating and uploading Breakpad symbols for OS libraries

Status: RESOLVED FIXED

(Camino Graveyard :: General, defect)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Rough cut

Stuart, can you look at this script (and also at the selected subset of Mac OS X frameworks and libraries)?

My shell-fu is mostly "copy what mento did in the feedhandlers Makefile", but this appears to work for me.  Ted apparently had a script at one time, but he can't find it now; I figure we may as well land this in camino/scripts/ or camino/tools/ so that there will be a record somewhere of how I implemented it.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Comment on attachment 376599 [details]
Rough cut

In thinking about it more, it's probably is better to include all the child frameworks in the meta-frameworks, since you (I) can't tell from header #includes whether or not the code in question uses just the subset of child frameworks I've seen in crashes reports or if other children might also be in use.

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Updated version

Here's an updated version that does comment 1, fixes a typo I found, and adds a few more things we probably want, based on additional data from Talkback.

I moved the files with spaces in their names to a separate section; we can probably hack around the bug in symbolstore.py, but I'm not quite there yet (the tricky part is going to be getting the UUID off of the first line and into the directory name where the .sym file needs to end up).

Comment 3

[Stuart Morgan]

Comment on attachment 377040 [details]
Updated version

Nice! Generating that list could not have been fun--and I should have thought of otool (see below) earlier, sorry about that :(

>SYMBOLSTORE_PY_PATH=$HOME/Camino/dev/trunk/mozilla/toolkit/crashreporter/tools/symbolstore.py
>DUMP_SYMS_PATH=$HOME/Camino/dev/trunk/CaminoStaticTrunk/camino/google-breakpad/src/tools/mac/dump_syms/build/Release/dump_syms
>SYMBOL_OUTPUT_DIR=$HOME/Camino/dev/trunk/macosx-$OS_VERSION-symbols
>SYMBOL_ARCHIVE_BASENAME=Mac_OS_X-$OS_VERSION

While we don't want to over-engineer this, I think we do want to make it more generic so that others could run it without having to edit the script. Luckily, that's pretty easy :)

-----
if [ $# -ne 3 ]; then
  echo "Usage: $0 <symbolstore.py path> <dump_syms path> <output directory>"
  exit 1
fi

SYMBOLSTORE_PY_PATH="$1"
DUMP_SYMS_PATH="$2"
OUTPUT_DIR="$3"
SYMBOL_OUTPUT_DIR="$OUTPUT_DIR/macosx-$OS_VERSION-symbols"
-----

The downside is more typing when you run it, but you can easily make a tiny convenience script for yourself that calls this script with all the right paths for you.

Alternately the first two arguments could be the source root and the objdir, but I think the clarity here is worth the extra path length... especially since you'll be using a wrapper script anyway ;)

># This list of files is generated based on mxr results for "-framework" in 
># mozilla/, frameworks Camino links against, common hits in Camino breakpad
># reports, as well as a few cherry-picked forward-looking files (CoreText).
># AE and LaunchServices switched frameworks in Mac OS X 10.5.

Sadly, it doesn't end there. First, we slipped a few libraries in with OTHER_LDFLAGS in Camino: libm and libpthread. The much nastier part though is that starting from this list, a thorough list would be the transitive closure of all frameworks and libraries that are linked starting from the list you have. You can see what a library loads with otool -L; to pick one at random (with versions removed from the output for brevity:

$ otool -L /System/Library/Frameworks/QuartzCore.framework/QuartzCore
/System/Library/Frameworks/QuartzCore.framework/QuartzCore:
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
	/usr/lib/libobjc.A.dylib
	/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
	/usr/lib/libffi.dylib
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
	/usr/lib/libxml2.2.dylib
	/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
	/usr/lib/libgcc_s.1.dylib
	/usr/lib/libSystem.B.dylib
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation

However: there's an argument to be made that we may not need all those symbols, since what's most interesting is what we call, not the full chain down into the depths of other libraries.

Obviously doing this by hand would be soul-crushing, so lets do this: go ahead with the list you have, but file a bug (and assign it to me, I suppose) to make a script that will start with a list of binaries and generate the complete list programatically. I'll hopefully get to it at some later date; strangely (sadly?), that will not be the first time I've written a script to do this...

>if [ -n "`echo $OS_VERSION | grep 10_4`" ]
>  then SYMBOL_FILES="$COMMON_OS_FILES $MACOSX_10_4_OS_FILES"
>  elif [ -n "`echo $OS_VERSION | grep 10_5`" ]
>  then SYMBOL_FILES="$COMMON_OS_FILES $MACOSX_10_5_OS_FILES"
>fi

The elif should move over to the left. Also, I'm a fan of the "; then" style of shell conditionals (as demonstrated above), as it make the body easier to follow if more things are added.

>#echo $SYMBOL_FILES

This was for temporary debugging, I assume? If so, it should go.

>python $SYMBOLSTORE_PY_PATH -a "ppc i386" -s / --vcs-info $DUMP_SYMS_PATH $SYMBOL_OUTPUT_DIR \
>  $SYMBOL_FILES > $SYMBOL_OUTPUT_DIR/$SYMBOL_ARCHIVE_BASENAME-symbols.txt

A good rule of thumb when dealing with variables in paths is 'quote everything'. It really sucks when a script dies when it's given paths with spaces ;) So this would be:

python "$SYMBOLSTORE_PY_PATH" -a "ppc i386" -s / --vcs-info "$DUMP_SYMS_PATH $SYMBOL_OUTPUT_DIR" \
  $SYMBOL_FILES > "$SYMBOL_OUTPUT_DIR/$SYMBOL_ARCHIVE_BASENAME-symbols.txt"

Also, I presume we don't want --vcs-info, and probably not the '-s /' either.

>#python /Users/smokey/Camino/[...]

More debugging cruft.

>cd $SYMBOL_OUTPUT_DIR && zip -r9D ../crashreporter-symbols-$SYMBOL_ARCHIVE_BASENAME.zip .

Quoting again.

># $(topsrcdir)/toolkit/crashreporter/tools/upload_symbols.sh $(SYMBOL_OUTPUT_DIR)/../crashreporter-symbols-$(SYMBOL_ARCHIVE_BASENAME).zip

Cruft again ;)

>echo "Mac OS X $OS_VERSION symbol archive generated in $SYMBOL_OUTPUT_DIR/../crashreporter-symbols-$SYMBOL_ARCHIVE_BASENAME.zip"

With the change from back at the top, you can use $OUPUT_DIR/[...] and get prettier output. Also, let's go ahead and make a variable for the zip file name so you can use it in both the command and the echo so they stay in sync if changed.

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Six less boneheaded errors; success!

Er, what we did for the spaces actually works in its, once I fixed two bone-headed errors in the SPACES_OS_FILES and supporting code: all those Input Methods are .apps, and inside the for loop, we want "$file" (we also don't want any ' around those filenames).

Once I fixed up some other path assumptions from the test script, I have a working script.

I am, however, seeing some oddness now in a couple of the child frameworks; even though I'm specifying only the foo.framework/foo file by name, I'm getting foo and foo_profile and/or foo_debug symbols.  It's like there's an * at the end somewhere.

Probably also need to double-check that all the explicitly-mentioned Frameworks are being dumped; offhand, it doesn't look like the numbers match, but it's late and my brain has already iterated several times through fixing the six errors that prevented the script from working at all, so who knows.

(Note to self when filing the otool bug: libm and libpthread -> libSystem -> libSystem.B)

Comment 5

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Comment on attachment 377099 [details]
Six less boneheaded errors; success!

Er, I still have one path wrong in the "spaces" section, because the filenames in Mac_OS_X-10_5_6-symbols.txt are coming out 
./macosx-10_5_6-symbols/Flash Player/77CB5AC61C456B965D0B41361B3F6CEA0/Flash Player.sym

When I'm not tired, I'll figure out where I need to split out the shorthand into two separate variables. :P

Comment 6

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Comment on attachment 377099 [details]
Six less boneheaded errors; success!

Clearing this request; I'll adapt this to use with our forked symbolstore.py and verify that all the explicit mentioned frameworks are getting dumped.

Comment 7

[Stuart Morgan]

Attachment: WIP, ruby version

After discovering issues in the shell script approach when the file list had files with spaces, I decided to just translate it to ruby (especially since I'll want it in an actual scripting language eventually for otool-fu). My ruby is really rusty, so I won't claim it's good, but at least it works with spaces ;)

I did a bit of trickery to have the file lists as here-docs rather than arrays just because it makes them easier to maintain (no need to stick " and ", around each line). I considered separate files, but since the lists will hopefully be going away in favor of dynamic generation that seemed like overkill.

Comment 8

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Final script, as landed in camino/tools

Just for the record, here's the final script we landed in camino/tools.

Comment 9

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

I forgot to add the comment I had locally in the shell version that spaces support requires our forked symbolstore.py, but all potential users for the foreseeable future know that.