387555 - should get automatic breakpad stacks from test crashes on unit test machines

Status: RESOLVED DUPLICATE of bug 481732

(Release Engineering :: General, enhancement)

Description

[(not currently active) Ted Mielczarek]

robcee did some work on this previously, but he says it's still a manual process.  At this point, we should be able to get stacks locally from crashes on the unit test boxes.  We will need:
1) The boxes to build with debug info, see the mozconfigs from the nightly boxes
2) A local copy of the breakpad minidump_stackwalk tool (this can be built once per platform and just copied over)
3) Set MOZ_CRASHREPORTER_NO_REPORT in the environment

Then when a crash occurs, we should be able to:
1) Locate the minidump file in $PROFILE/minidumps
2) Run |make buildsymbols| in the objdir, which will produce a $objdir/dist/crashreporter-symbols/$BUILD_ID directory containing symbols
3) Run |minidump_stackwalk $minidump $objdir/dist/crashreporter-symbols/$BUILD_ID|, which will produce a stack trace.

Comment 1

[(not currently active) Ted Mielczarek]

From a short conversation, I think we'll either need to build with MOZILLA_OFFICIAL=1, or set MOZ_CRASHREPORTER=1 in the environment, otherwise breakpad crash reporting will not be active.

Comment 3

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

(For the Linux case, we can get there quick-and-dirty by re-enabling the oh_crap handler and building with symbols.  Might be a worthwhile interim measure, especially given the periodic mochicrashes on qm-centos5-01.)

Comment 4

[Rob Campbell [:rc] (:robcee)]

1) how much time does this add to the build cycle?

2) is this oh_crap handler documented somewhere?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

roc was asking for something similar on IRC recently, and I agree this sounds like a useful thing to do. 

One issue to solve is that we compile, run make check, and then remove all profiles before running reftest, crashtest and so on; so it's likely that any minidumps would be deleted in the case of an intermittent crash, just from the delay between a failure and a request to get a stack. We could rsync them somewhere else before hand.

On Windows I think we can configure the Visual Studio debugger to dump a stack straight into the log, which has obvious advantages. Anyone got any tricks for doing that on Linux & Mac too ? If that can be done without intruding on other builds (on the same box) then it might be a more automatic way to go.

Comment 6

[(not currently active) Ted Mielczarek]

We could pretty easily add some functionality to Mochitest to check for a minidump in the profile folder if the app exits with an error, and have an environment variable set to the path of minidump_stackwalk, which we could run on it. Once bug 468913 is fixed, we could probably do the same for reftest/crashtest. TUnit is a bit tougher, because the Breakpad handler doesn't get set in xpcshell.

Comment 8

[[e-mail] (:bhearsum)]

I don't know where this falls on the overall priority list but we're certainly not getting to it this year.

Comment 9

[Damon Sicore (:damons)]

I get two or three requests for this a week now.  Seems like we should bump this up in priority.

Comment 10

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

We have more disk on the socorro machine now too, so keeping the symbols for 48 hours or whatever shouldn't be a problem.

Comment 11

[Damon Sicore (:damons)]

Example issue that fixing this would help:  bug 473013

Comment 12

[(not currently active) Ted Mielczarek]

(In reply to comment #10)
> We have more disk on the socorro machine now too, so keeping the symbols for 48
> hours or whatever shouldn't be a problem.

That's not relevant to getting stacks from the unit test machines, as they currently do their own builds. It is relevant to getting stacks from Talos, which is a separate bug.

Comment 13

[(not currently active) Ted Mielczarek]

bsmedberg fixed this in bug 481732.

Comment 14

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.