530592 - Inlined static functions seems to confuse windows stack walking code.

Status: RESOLVED WONTFIX

(Toolkit :: Crash Reporting, defect)

Description

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

It's very common that opening a minidump in Visual Studio gives a much better stack than when looking at the socorro web interface.

Many times the socorro stack contains much fewer frames than Visual Studio is able to produce, but I've also seen cases where the socorro stack is simply missing individual frames.

In particular it seems that the agressive optimization that MSVC does since we use PGO is confusing socorro.


Would it be possible to call into the visual studio toolchain somehow? That might make things more future proof too in case microsoft adds additional optimizations in future MSVC releases.

Comment 1

[(not currently active) Ted Mielczarek]

This bug is kind of too general to be useful. We can fix specific cases, but not a "stack walking sucks" bug.

In particular, some of what you're talking about might be covered by bug 526484, bug 529266, bug 522701 and bug 529363.

If you find a specific crash report where we're producing a bad stack, please file a bug on that, and grab the minidump so we can test and fix any bugs we find. We've fixed bugs with the stack walker itself (bug 519616) as well as with the debug info our builds were producing (bug 524256).

To answer your final question, we are using the DIA SDK at build-time, Microsoft's supported API for reading debug information out of PDB files. Doing this at processing time would require us to run the processor on a Windows machine, which is not a path we want to go down. I don't think there are any fatal flaws in our current approach, just small fixable bugs that we're only noticing now due to increased scrutiny of crash reports.

Finally, there will always be confusing stacks due to compiler optimization, it's just a fact of life. MSVC's PGO optimizer does aggressive inlining, virtual call speculation, etc. Not even their debugger can get you the expected stack out of that, so it's just something you have to live with.

Comment 2

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

You beat me to finding examples :(

Here is one example where socorro is just able to get three frames, but visual studio is giving me a full crash stack.

http://crash-stats.mozilla.com/report/index/1c63a802-c55f-4db1-b915-fa58b2091106

In this example socorro stops at the point where MSVC inlined a couple of functions. Even MSVC isn't able to display the "inlined away" frames. However it is able to keep walking up the stack.

Given that we apparently don't want to use MSVCs code to do the actual stack walking, I'll try to file individual bugs when I see other patterns that confuse socorro. Making this bug be about specifically the inlining confusing socorro.

Comment 3

[(not currently active) Ted Mielczarek]

Ok, that's much more useful! Can you grab a minidump that matches this signature and email it to me? I can debug the stackwalker locally with it.

Comment 4

[(not currently active) Ted Mielczarek]

And please do file separate bugs for other issues you find. We can always dupe them if they turn out to be the same issue later, but combining a bunch of issues into one bug always results in confusion, as I'm sure you're aware.

Comment 5

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Sent minidump over email

Comment 6

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

I should mention that in this minidump not even MSVC is able to give a *full* stack. However it's definitely getting further than socorro is (and it so happened that it got as far as I needed to fix the bug).

So there's probably additional weirdness up the stack, possibly traced code. But that's for a separate bug.

Comment 7

[(not currently active) Ted Mielczarek]

Can you paste the stack that MSVC gives you here, just so I have something to compare with?

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Attachment: MSVC stack

Adding as attachment instead since the format from MSVC is so messy and I don't want to remove information in case it's useful to you.

Comment 9

[Laura Thomson :laura]

Priority, Jonas?

Comment 10

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Hard to say as I don't know what else is on your list and so have nothing to gauge against.

As I recall it, the result of this is that much of the time we end up with shorter stacks than we could, generally resulting in several different crashes getting grouped together into one socorro signature.

It also makes diagnosing through the socorro interface much harder. Not a huge deal, but it does mean that you generally can't use the socorro interface to figure out who is responsible for a given crash.

Comment 11

[(not currently active) Ted Mielczarek]

We can move all these stack walking bugs to the Toolkit:Breakpad Integration component, since they're really Breakpad bugs.

Comment 12

[Robert Kaiser]

Moving to Toolkit:Breakpad Integration per comment #11 (wondering why Ted didn't do that himself when writing this).

Comment 13

[Gabriele Svelto [:gsvelto]]

The minidump exhibiting this behavior is long gone and we now support inlined functions. We also have more recent bugs regarding stack-walking issues specific to the new stack walker, closing.