360327 - Steps to a working breakpad crash reporting system

Status: RESOLVED FIXED

(Core Graveyard :: Tracking, defect)

Description

[(not currently active) Ted Mielczarek]

vlad asked for a bug listing what we need to get our airbag-based crash reporting into a usable state, at least for our win32 builds.

1) We need to work on the server side of things.  I filed bug 358302 for setting up a server for this, but the software needs to be finished first.  We will need a better collector, and also a way to query the database of reports.
2) We need to be able to collect symbols from tinderbox builds.  I filed bug xxx for integrating the airbag dump_syms tool into the build process, which will get us the symbols in the proper format for airbag's processing tools, but we will also need the tinderbox integration for copying those symbols to the symbol server.

I think these two issues are the only things blocking deployment of our airbag system.  The symbol collection/tinderbox integration should be pretty easy.  The server software I have some work done on, and some Seneca students have expressed interest in working on, so I think we're close there.  There are obviously lots of improvements we can make once we get the base system working, but I don't think they're necessary for a first cut.

Comment 1

[(not currently active) Ted Mielczarek]

I really did intend to fill in that second bug number up there, bug 359151.

Comment 2

[Jonathan Watt [:jwatt]]

Great stuff. I know this is largely mozilla.org setup stuff but, as much as possible, could all this work be done in public? By that I mean can all steps taken, problems encountered, lessons learned be reported please? Those of us who are building applications on the XULRunner platform will find such information very useful I think. It would be good to know who the individuals doing the server work are too, assuming they don't mind being asked a few questions about the process.

Comment 3

[chris hofmann]

cc'ing preed.  any advantage in using the same symbol copy mechanism that talkback uses in the build scripts, and just expanding this to point to a new set of servers for airbag?

Comment 4

[Benjamin Smedberg]

We are actively trying to move as much of the build logic into the makefiles and out of the tinderbox scripts as possible. Now that talkback and its build process are open, it is at least theoretically possible to build and stage a release entirely from makefiles. So no, I don't think we should be trying to reuse the existing symbol staging scripts.

Comment 5

[J. Paul Reed [:preed]]

(In reply to comment #3)
> cc'ing preed.  any advantage in using the same symbol copy mechanism that
> talkback uses in the build scripts, and just expanding this to point to a new
> set of servers for airbag?

I think we should borrow some of the concepts (a script that can re-upload the symbols in case of failure, etc.) used in the current scripts, but the current scripts are kinda convoluted.

The symbol upload script is created by makefiles upon each build, and then run from those makefiles.  I don't see a reason to not just create a script that understands how to find and upload symbols.

Not sure if this should be part of the make machinery, since that complicates being able to just re-run symbol uploads. It's really nice to be able to go into a tinderbox days after a build was run, and re-upload all the symbols because the talkback server ran out of disk space or crashed or the tinderboxen didn't have the right ssh keys, etc., etc., etc.

Comment 6

[(not currently active) Ted Mielczarek]

For the benefit of those of us who don't know anything about how Talkback works behind the scenes, can you briefly explain how symbols get to the symbol server?  You might also want to drop a note on bug 359151, since the implementation there will directly impact that process.

Comment 7

[J. Paul Reed [:preed]]

(In reply to comment #6)
> For the benefit of those of us who don't know anything about how Talkback works
> behind the scenes, can you briefly explain how symbols get to the symbol
> server?  You might also want to drop a note on bug 359151, since the
> implementation there will directly impact that process.

You can see snippets of the current code/process in bug 344466.

The upshot is that a Makefile creates a shell script and dumps that shell script into a dated directory with all the symbols to upload. It then attempts to run this script.

The nice part of using this methodology is that if the process fails for some reason, a person can go back and re-run the shell script in the dated directory to get the same effect.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

What will those of us who create our own builds

Comment 9

[Marc Bejarano]

all bugs that this depends on have been closed.  should this be closed or are there missing dependencies?

Comment 10

[(not currently active) Ted Mielczarek]

This is still blocked by getting the database and processor setup on the staging servers.  The work to get them to that state is happening over on the Socorro project page: http://code.google.com/p/socorro/ .  We'll file IT deployment bugs as blockers on this once the code is ready.

Comment 11

[(not currently active) Ted Mielczarek]

It's alive!