Deploy new minidump_stackwalk to solve stackwalking issues with MSVC2010



7 years ago
7 years ago


(Reporter: ted, Assigned: rhelmer)


Dependency tree / graph

Firefox Tracking Flags

(Not tracked)



(4 attachments, 1 obsolete attachment)

Since we switched to VC++ 2010, we've been seeing some bad stacks in certain cases. It looks like the new compiler is producing stack unwind information that Breakpad doesn't know how to handle. There are two specific things that are causing problems:
1) A new operator in the unwind expressions: @. This appears to be an alignment operator.
2) A new built-in value: .raSearch. We haven't narrowed down exactly what the difference is between this and .raSearchStart.

I have a WIP Breakpad patch that adds support for both of these, but I'm not sure if it's correct, and if it's going to fix the problem.
Created attachment 596661 [details] [diff] [review]
Add support for @ operator and .raSearch in Breakpad x86 stackwalker

Very much a WIP. I don't know if the @ implementation is correct, and I'm pretty sure the value of .raSearch isn't perfect.
I put my patch for the @ operator up for review upstream:

bsmedberg has a patch to make .raSearch handling work which he's also going to upstream.
Assignee: nobody → benjamin

Comment 3

7 years ago
The .raSearch fix is up at

I need the @ patch to land before I can upload a useful patch for the remaining issue, since it touches/refactors the same code blocks.

Comment 4

7 years ago
Created attachment 597449 [details] [diff] [review]
Postfix tokenizer fix, on top of the "@" change

For archiving.
Attachment #596661 - Attachment is obsolete: true

Comment 6

7 years ago

This bug is now ready to morph into a deployment bug. We need minidump_stackwalk recompiled against breakpad r927 and deployed to staging.
Assignee: benjamin → server-ops
Component: General → Server Operations: Web Operations
Product: Socorro →
QA Contact: general → cshields
Version: unspecified → other


7 years ago
Summary: Stackwalking issues with MSVC2010 → Deploy new minidump_stackwalk to solve stackwalking issues with MSVC2010

Comment 7

7 years ago
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #6)
> This bug is now ready to morph into a deployment bug. We need
> minidump_stackwalk recompiled against breakpad r927 and deployed to staging.

We now have a jenkins job for breakpad:

The Socorro job automatically pulls the last successful build (which is that one):

So I think we don't need to do anything special here, this should automatically go out on the next Socorro release (and should be on the dev server already)

Comment 8

7 years ago
rhelmer is going to help shepherd this onto stage so that it can go out on Wednesday with the scheduled release.
Assignee: server-ops → rhelmer

Comment 9

7 years ago
Created attachment 601042 [details]
Dump for testing on stage

I'm going to attach a .dump and .extra file to test this with.

Comment 10

7 years ago
Created attachment 601043 [details]
.extra file for testing on stage


7 years ago
Attachment #601043 - Attachment mime type: application/xml → application/octet-stream


7 years ago
Component: Server Operations: Web Operations → Backend
Product: → Socorro
QA Contact: cshields → backend
Target Milestone: --- → 2.4.4

Comment 11

7 years ago
Created attachment 601403 [details]
json file for testing on stage

This is equivalent to the .extra file, but formatted as JSON so we can use the submitter tool.

Comment 12

7 years ago
I inserted crashes but can't complete testing quite yet, we're doing a DB import on dev and staging hbase seems to be down (bug 730986).
Depends on: 730986

Comment 13

7 years ago
OK here is an example:

Can someone help to make sure this looks ok? I'd like to also make sure "normal" jobs work before marking verified.
Last Resolved: 7 years ago
Resolution: --- → FIXED

Comment 14

7 years ago
KaiRo says comment 13 looks ok, and seems to do no harm with other crashes:

Comment 16

7 years ago
Are you sure the symbols were present at processing time? Without actually having the processing log, this looks like the symbols weren't fully uploaded/synced yet.

Comment 17

7 years ago
Oh, ignore me, the google_breakpad symbols are in xul.dll also. I don't know.
Kairo: those two crashes you linked are both 64-bit (note the amd64). We don't currently have useful unwind info in Win64 symbols. That's bug 548035.
You need to log in before you can comment on or make changes to this bug.