Closed Bug 587073 Opened 12 years ago Closed 10 years ago

Symbol package uploads duplicate dSYM files

Categories

(Toolkit :: Crash Reporting, defect)

x86
macOS
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla16

People

(Reporter: ted, Assigned: ted)

References

Details

Attachments

(1 file)

On OS X we currently package the .dSYM files with the Breakpad symbols and upload them to the symbol server. Unfortunately right now the .dSYM files are universal for our universal builds (they contain both ppc and x86), but we upload two copies of each. This is lame. We should instead be running "dsymutil --arch=ARCH".
This is causing us to use 616MB(!) per symbol upload on the symbol store. (Each copy of XUL.dSYM.tar.bz2 is 254MB.)
Blocks: 598757
This is responsible for about 300GB of extra disk usage, I estimate.
(In reply to comment #0)
> universal for our universal builds (they contain both ppc and x86), but we
> upload two copies of each. This is lame. We should instead be running "dsymutil
> --arch=ARCH".

http://mxr.mozilla.org/mozilla-central/source/Makefile.in#151
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#710

makes it look to me like you're passing in two archs and translating -a foo to the --arch=foo syntax dsymutil is expecting.  What am I missing?  Or are you actually running symbolstore.py against each thin binary (I thought that was only something we had to do in Camino to work around assumptions in the build system, and other Gecko apps acted on the fat binary)?
We pass it all the arches, so we call "dsymutil --arch=ppc --arch=i386 whatever", thus getting a universal .dSYM. And then we copy it to the symbol dir twice, because we loop over each arch and handle copy_debug inside there:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#554
Duplicate of this bug: 753800
(In reply to Ted Mielczarek [:ted] from comment #1)
> This is causing us to use 616MB(!) per symbol upload on the symbol store.
> (Each copy of XUL.dSYM.tar.bz2 is 254MB.)

It's almost 1GB, now.
Assignee: nobody → ted.mielczarek
Okay, I've run this past tryserver and it seems to work. The meat of this patch is just changing ProcessFile to use enumerate for looping over archs, and only calling CopyDebug for the first architecture. The rest is mostly tweaks so the unit tests work properly.
Attachment #634103 - Flags: review?(nrthomas)
Comment on attachment 634103 [details] [diff] [review]
stop uploading duplicate dSYM files in the symbol package

>diff --git a/toolkit/crashreporter/tools/symbolstore.py b/toolkit/crashreporter/tools/symbolstore.py
>         print >> sys.stderr, "Processing file: %s" % file
>+        sys.stderr.flush()

Oh yes!

>-                    if self.copy_debug:
>+                    # only copy debug the first time if we have multiple architectures
>+                    if self.copy_debug and arch_num == 0:
>                         self.CopyDebug(file, debug_file, guid)

Was hoping we could short circuit processing both XUL files for a big time win, but see they yield different sym files. Oh well.

>diff --git a/toolkit/crashreporter/tools/unit-symbolstore.py b/toolkit/crashreporter/tools/unit-symbolstore.py

All the stubbing out made my brain hurt, so I'm relying on that try run for this.
Attachment #634103 - Flags: review?(nrthomas) → review+
Right, since it's a universal binary we really do need to dump both architectures. This was the least invasive way I found to accomplish what I wanted to do.

Thanks for the reviews!
https://hg.mozilla.org/mozilla-central/rev/38fc0b6b4e62
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla16
Looks like a nice savings (~400MB per build):
$ ~/totalsize.py ./firefox-16.0a1-Darwin-20120620030535-x86-macosx64-symbols.txt 
total: 1069482711 bytes (1019.94MB, 1.00GB)

$ ~/totalsize.py ./firefox-16.0a1-Darwin-20120621030536-x86-macosx64-symbols.txt
total: 620038212 bytes (591.31MB, 0.58GB)
Tiny followup because I lost the bit where we redirected the stdout of dsymutil to /dev/null:
http://hg.mozilla.org/integration/mozilla-inbound/rev/32a058c88ce8
You need to log in before you can comment on or make changes to this bug.