815695 - Pull symbols from Apple updates semi-automatically

Status: RESOLVED FIXED

(Socorro :: Symbols, task)

Description

[Benjamin Smedberg]

<ted> if you're interested in pursuing this, mrmiller (was hanging out here previously) had a concept setup to fetch updates from apple's update server
<ted> and dump them locally
<ted> we'd probably need some mac hardware setup to run it on
<ted> i haven't ever had cycles to look into that, but it should produce pretty good results
<ted> the updates contain the full binaries of what's being updated
<ted> so you can just unwrap them and run dump_syms
<ted> there's open-source update client software out there already, he was using that

We should try to automate this so that we don't have to manually upload symbols as in bug 815543.

Comment 1

[michaelrmmiller+bugzilla]

I put some work into Python scripts to automate the process of unpacking the installer packages and running dump_syms. I also looked into getting things running on a Linux box instead of Mac hardware. I still think it's possible, but I forget the precise state I left things in when I had to move on to other tasks. My recollection was I was having trouble mounting the DMGs on Linux, but if that part is working, it should all work.

A quick survey of the tools I was going to use:

Reposado
https://github.com/wdas/reposado
Cross-platform open source client for Apple's update servers.

Required tools for Linux:
    pax
    gzip
    tar
    xar (http://code.google.com/p/xar/)
    xpwn's dmg (https://github.com/planetbeing/xpwn)

PackageSymbolDumper.py (attached)
Mostly implemented script to temporarily unpack a package and pull the symbols from a large number of executables found within. This probably hits too many files, but seeing how the list of needed dylibs changes from version to version, I figured the space cost wasn't worth the hassle of being more selected.

DumpBreakpadSymbols.py (attached)
A simple script to walk a directory tree and dump symbols for any executables found within.

Comment 2

[michaelrmmiller+bugzilla]

Attachment: Unpacks a Mac OS update installer package and dumps the symbols within.

Comment 3

[michaelrmmiller+bugzilla]

Attachment: Script to dump Breakpad symbols for all executables in a directory

Comment 4

[(not currently active) Ted Mielczarek]

I could not ever find this bug for some reason! I did follow up on this a bit:
https://github.com/luser/breakpad-mac-update-symbols

I wrote a Dockerfile to wrap these scripts up, and I was able to run it in Taskcluster:
https://tools.taskcluster.net/task-graph-inspector/#XhNHyG__SryUwmHpD1SCJA

It needs a little more work to be able to be run on a regular schedule to keep the symbols up to date.

Comment 5

[(not currently active) Ted Mielczarek]

BenWa did some work in bug 1296728 to fix the scripts up to handle modern OS X update packages (Apple switched to using a new payload format at some point). I'm going to call this FIXED, and I'll file a new bug to get it up and running on a regular basis.