587073 - Symbol package uploads duplicate dSYM files

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[(not currently active) Ted Mielczarek]

On OS X we currently package the .dSYM files with the Breakpad symbols and upload them to the symbol server. Unfortunately right now the .dSYM files are universal for our universal builds (they contain both ppc and x86), but we upload two copies of each. This is lame. We should instead be running "dsymutil --arch=ARCH".

Comment 1

[(not currently active) Ted Mielczarek]

This is causing us to use 616MB(!) per symbol upload on the symbol store. (Each copy of XUL.dSYM.tar.bz2 is 254MB.)

Comment 2

[(not currently active) Ted Mielczarek]

This is responsible for about 300GB of extra disk usage, I estimate.

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #0)
> universal for our universal builds (they contain both ppc and x86), but we
> upload two copies of each. This is lame. We should instead be running "dsymutil
> --arch=ARCH".

http://mxr.mozilla.org/mozilla-central/source/Makefile.in#151
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#710

makes it look to me like you're passing in two archs and translating -a foo to the --arch=foo syntax dsymutil is expecting.  What am I missing?  Or are you actually running symbolstore.py against each thin binary (I thought that was only something we had to do in Camino to work around assumptions in the build system, and other Gecko apps acted on the fat binary)?

Comment 4

[(not currently active) Ted Mielczarek]

We pass it all the arches, so we call "dsymutil --arch=ppc --arch=i386 whatever", thus getting a universal .dSYM. And then we copy it to the symbol dir twice, because we loop over each arch and handle copy_debug inside there:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#554

Comment 6

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted] from comment #1)
> This is causing us to use 616MB(!) per symbol upload on the symbol store.
> (Each copy of XUL.dSYM.tar.bz2 is 254MB.)

It's almost 1GB, now.

Comment 7

[(not currently active) Ted Mielczarek]

Attachment: stop uploading duplicate dSYM files in the symbol package

Okay, I've run this past tryserver and it seems to work. The meat of this patch is just changing ProcessFile to use enumerate for looping over archs, and only calling CopyDebug for the first architecture. The rest is mostly tweaks so the unit tests work properly.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 634103 [details] [diff] [review]
stop uploading duplicate dSYM files in the symbol package

>diff --git a/toolkit/crashreporter/tools/symbolstore.py b/toolkit/crashreporter/tools/symbolstore.py
>         print >> sys.stderr, "Processing file: %s" % file
>+        sys.stderr.flush()

Oh yes!

>-                    if self.copy_debug:
>+                    # only copy debug the first time if we have multiple architectures
>+                    if self.copy_debug and arch_num == 0:
>                         self.CopyDebug(file, debug_file, guid)

Was hoping we could short circuit processing both XUL files for a big time win, but see they yield different sym files. Oh well.

>diff --git a/toolkit/crashreporter/tools/unit-symbolstore.py b/toolkit/crashreporter/tools/unit-symbolstore.py

All the stubbing out made my brain hurt, so I'm relying on that try run for this.

Comment 9

[(not currently active) Ted Mielczarek]

Right, since it's a universal binary we really do need to dump both architectures. This was the least invasive way I found to accomplish what I wanted to do.

Thanks for the reviews!

Comment 10

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/38fc0b6b4e62

Comment 11

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/38fc0b6b4e62

Comment 12

[(not currently active) Ted Mielczarek]

Looks like a nice savings (~400MB per build):
$ ~/totalsize.py ./firefox-16.0a1-Darwin-20120620030535-x86-macosx64-symbols.txt 
total: 1069482711 bytes (1019.94MB, 1.00GB)

$ ~/totalsize.py ./firefox-16.0a1-Darwin-20120621030536-x86-macosx64-symbols.txt
total: 620038212 bytes (591.31MB, 0.58GB)

Comment 13

[(not currently active) Ted Mielczarek]

Tiny followup because I lost the bit where we redirected the stdout of dsymutil to /dev/null:
http://hg.mozilla.org/integration/mozilla-inbound/rev/32a058c88ce8

Comment 14

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/32a058c88ce8