683162 - Get a new minidump_stackwalk into production to fix stackwalk errors

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[(not currently active) Ted Mielczarek]

We're seeing crashes showing up with bad stacks, see bug 677579 and bug 677580. The root cause here is that Breakpad is misinterpreting the exception context, so it is failing to walk the stack of the crashing thread properly. I need to patch upstream Breakpad and then we can get an updated minidump_stackwalk in production.

I'm not sure what changed to make this break, perhaps a new version of DbgHelp.dll?

Comment 1

[(not currently active) Ted Mielczarek]

Patch up for review upstream:
http://breakpad.appspot.com/302001

Comment 2

[(not currently active) Ted Mielczarek]

Landed upstream:
http://code.google.com/p/google-breakpad/source/detail?r=831

Comment 3

[(not currently active) Ted Mielczarek]

rhelmer says this will get pulled in automatically on the next build, since we are building minidump_stackwalk from Breakpad tip.

To verify this fix, we should be able to reprocess some of the signatures from the bugs in comment 0 and verify that they get new signatures and sane-looking stacks.

Comment 4

[(not currently active) Ted Mielczarek]

This didn't get rolled out because I also convinced rhelmer to start running the Breakpad unit tests, and they hang, so the build never made it to production.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Ted Mielczarek [:ted, :luser] from comment #4)
> This didn't get rolled out because I also convinced rhelmer to start running
> the Breakpad unit tests, and they hang, so the build never made it to
> production.

I think we also skipped the verification step (comment 3), I can do it this time and will work with QA so they know how to do this in the future (not their fault, I think this is something we've never set QA up for).

Comment 6

[Matt Brandt [:mbrandt]]

(In reply to Robert Helmer [:rhelmer] from comment #5)> 
> I think we also skipped the verification step (comment 3), I can do it this
> time and will work with QA

Thanks rhelmer: I'll look forward to understanding the verification steps.

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to Matt Brandt [:mbrandt] from comment #6)
> (In reply to Robert Helmer [:rhelmer] from comment #5)> 
> > I think we also skipped the verification step (comment 3), I can do it this
> > time and will work with QA
> 
> Thanks rhelmer: I'll look forward to understanding the verification steps.

Cool thanks, I went ahead and verified on dev (which pulls master), I'll let you do these when we get stage updated :) Feel free to grab me if you need any help:

OK generated new socorro-release and socorro-master builds:
https://jenkins.mozilla.org/job/socorro-release/7/
https://jenkins.mozilla.org/job/socorro-master/44/

Verified this on crash-stats-dev (which tracks and auto-installs master). Here are the steps:

1) identify bad signature (kernelbase.dll@0x10db)

2) download raw dump/json from instance of above https://crash-stats.mozilla.com/report/index/1c11af84-3fb7-4196-a864-cf0622110911

3) submit this json/dump to crash-stats-dev:

--
# /bin/sh
git clone https://github.com/mozilla/socorro
cd socorro
virtualenv socorro-virtualenv
. socorro-virtualenv/bin/activate
pip install poster
cp scripts/config/submitterconfig.py.dist scripts/config/submitterconfig.py
export PYTHONPATH=.:thirdparty
python scripts/submitter.py -u https://crash-reports-dev.allizom.org/submit -j ~/Downloads/1c11af84-3fb7-4196-a864-cf0622110911.json -d ~/Downloads/1c11af84-3fb7-4196-a864-cf0622110911.dump
--

4) search for returned CrashID on crash-stats-dev:
CrashID=bp-70e5870a-1f9d-491b-99f7-a9d1d2110913
https://crash-stats-dev.allizom.org/report/index/70e5870a-1f9d-491b-99f7-a9d1d2110913

5) verify signature is not "kernel.dll@random": 
ILFindLastID

Comment 8

[Matt Brandt [:mbrandt]]

rhelmer .. thx for wonderfully well written steps to qa. This will go directly into our little blackbook of qa awesomeness.

QA verified on dev. Waiting to verify on stage.