495464 - [tracking bug] Full crash data on talos and unittests

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P3)

Description

[Chris AtLee [:catlee]]

Our current minidumps aren't always sufficient to track down the source of crashes (see bug #495353).

We should be collecting full crash data, packaging that up with the build and debug symbols, and putting it somewhere where developers can examine it.

We need to continue to include the stack trace in the log; sometimes you can tell what's going wrong just from the stack trace.

Comment 1

[Chris AtLee [:catlee]]

For Windows:
http://www.debuginfo.com/articles/ntsdwatson.html

Comment 2

[(not currently active) Ted Mielczarek]

So, I think what we can do here is make Breakpad write a dump, then pass off to the OS exception handler, which can write a full dump/core/whatever. We already have a setting in our code (exposed only on OS X, at the moment) to do precisely this:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/nsExceptionHandler.cpp#167
see the "defaults write" bit here:
https://developer.mozilla.org/en/Environment_variables_affecting_crash_reporting

We could easily expose this on all platforms via an environment variable.

Comment 3

[Mike Shaver (:shaver emeritus)]

This would be a quantum leap in our ability to debug these sorts of crashes.  It is difficult to overstate the value.  Can I donate a kidney?

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Marking as future for now - seems like a pretty involved project and we haven't got the resources at the moment.  We are working on bits and pieces of this but we won't be able to roll this out any time soon.

Comment 5

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 6

[(not currently active) Ted Mielczarek]

See also bug 544062, which is about saving the minidumps from unittest runs for later inspection, which would get us partway to this.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to comment #6)
> See also bug 544062, which is about saving the minidumps from unittest runs for
> later inspection, which would get us partway to this.

This single bug is masking a lot of complexity, I think. I agree that the final outcome seems full of win, but I'm unsure how to get there from here. 

Ted, I guess these questions are for you: Once bug 544062 is resolved, what other steps are required? Are those steps different based on platform? 

Comment #2 seems to explain part of it, but it sounds like there is mozilla-central work that would need to happen first too.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #7)
> (In reply to comment #6)
> > See also bug 544062, which is about saving the minidumps from unittest runs for
> > later inspection, which would get us partway to this.
> 
> This single bug is masking a lot of complexity, I think. I agree that the final
> outcome seems full of win, but I'm unsure how to get there from here. 

I am going to turn this into a tracking bug for work to get this done for our three supported platforms, windows, osx and linux.

Comment 9

[Chris Cooper [:coop] (he/him)]

I don't really see the point of having this tracking bug now that we have the platform-specific ones, the Mac one having already been WONTFIXed.

I'm going to mine this bug for any platform-specific info, add it to the appropriate platform bugs, and then close this one.

Comment 10

[Chris Cooper [:coop] (he/him)]

Broken into smaller per-platform chunks in bug 640343 and 640345.