489019 - Upload symbols to Socorro.

Status: RESOLVED FIXED

(Camino Graveyard :: General, defect)

Description

[Stuart Morgan]

We have (some of) our symbols being generated now, so we need to post them.

Ted, is it just as simple as feeding all our dSYM files and all the core dylibs we ship into symbolstore.py, zipping it up, then running the uploader script? Do we need to follow the same enclosing folder hierarchy as the buildsymbols target does when zipping?

(Apparently I should have looked into this more before putting target-level scripts to generate breakpad symbol files, since this does it for us...)

Comment 1

[(not currently active) Ted Mielczarek]

I don't know if you can use "make buildsymbols" + "make uploadsymbols" in the top-level Makefile, they're what we use. "buildsymbols" will point symbolstore.py at dist/bin, run dsymutil on all the binaries it finds, then run dump_syms and zip the proper directory structure.
http://mxr.mozilla.org/mozilla-central/source/Makefile.in#180

If not, well, then just call symbolstore.py just like that, and zip the contents of the directory.

Comment 2

[Stuart Morgan]

Note to self: flip SendAndExit back to YES once we do this; we've turned it off until we have symbols so there's some way to get useful reports.

Comment 3

[Stuart Morgan]

Actually we'll remove SendAndExit, per bug 488518 comment 1.

Comment 4

[Stuart Morgan]

Ted, do you know what the process is for getting caminobld set up as a valid account (with key) on dm-symbolpush01.mozilla.org, and how to find out what our SYMBOL_SERVER_PATH should be?

Comment 5

[(not currently active) Ted Mielczarek]

Just file a Server Ops bug asking for that, and they can make it whatever you want. Firefox symbols go to /mnt/netapp/breakpad/symbols_ffx, Thunderbird are in symbols_tbrd, etc. CC aravind on the bug, as he's taken care of this in the past.

Comment 6

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #5)
> Just file a Server Ops bug asking for that

Filed bug 490184; thanks, Ted.

Comment 7

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

So, we have one more problem here.  Everyone else's generation+upload is done in post-mozilla-rel.pl (via commands to "make buildsymbols" + "make uploadsymbols" as noted above), and post-mozilla-rel.pl has this awesome logic ("$cachebuild") to determine whether we're a nightly/release or just any old build.  If we're a nightly/release, there be symbols; otherwise, no symbols.

After staring at several thousand lines of perl across a couple of files for a couple of hours (at, admittedly, a late hour), I can't see a way for us to get that information ("are we a nightly/release build?") into the environment where we can then make our uploadsymbols pseudo-target ifdefed on only then, rather than clogging the server with worthless symbols every 20 minutes all day.

Comment 8

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(The part 2 of that comment is that symbols only get uploaded for successful builds--those that succeed in unify and that pass tests--too.)

Comment 9

[Stuart Morgan]

Attachment: Camino build changes [landed]

Part I: this makes our symbol build and upload into separate targets. That way we can change post-mozilla-rel.pl in part II to make our build and upload targets instead of the core versions.

This also addresses bug 488512 comment 16, adding --vcs-info (and doing a bit of fixup since our paths aren't quite what the script expects).

Comment 10

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Comment on attachment 375372 [details] [diff] [review]
Camino build changes [landed]

r=ardissone

Comment 11

[Stuart Morgan]

Comment on attachment 375372 [details] [diff] [review]
Camino build changes [landed]

Landed on cvs trunk.

Comment 12

[Stuart Morgan]

I checked in a follow-up fix to Makefile.in; I was feed the wrong path in to upload_symbols.sh

Comment 13

[Stuart Morgan]

Ted, we appear to be missing some final bit of magic. We got symbols uploaded, e.g.:
  http://symbols.mozilla.org/symbols_camino/Camino/419A401095604F04A317E2105CEFC8F10/Camino.sym
and then I crashed that build:
  http://crash-stats.mozilla.com/report/index/9e20b6a1-4e62-45cf-81ed-703992090501?p=1
But as you can see from the crash report the symbols aren't being used for the Camino binary.

Do you know what we are missing? Something to do with the new symbols_camino path, perhaps?

(The only other difference I know of between our symbols and others is that we have two symbol manifests (if that's the term; the output of symbolstore.py) with a different naming schemee, but I assume that is purely for the benefit of humans rather than Socorro)

Comment 14

[(not currently active) Ted Mielczarek]

The symbols.txt files are purely for our symbol cleanup scripts. The processor looks symbols up by filename+UUID (you can verify they match in the Modules tab). You should ask aravind if he added your new path to the processor search path in bug 490184.

Comment 15

[Stuart Morgan]

Everything is working, and getting our local tinderbox script changes landed is now bug 491642, so I think we can call this FIXED.

Ted, thanks so much for all your help! We couldn't have gotten here with your patience with all my n00b questions ;)

Comment 16

[(not currently active) Ted Mielczarek]

I'm just glad to get everyone off that *other* crash reporting system finally!