466026 - raw dump would be much more useful with absolute or library-relative addresses

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'd like to be able to use the "Raw Dump" to figure out which instruction a crash happened on, rather than just which line.  This can be useful for figuring out what the problem is.

The problem is that the "raw dump" part of the stack trace looks like this:

0|0|xul.dll|nsStyleSet::AddImportantRules(nsRuleNode *,nsRuleNode *)|cvs:cvs.mozilla.org/cvsroot:mozilla/layout/style/nsStyleSet.cpp:3.190|439|0x1e

The very last number there looks like it's the offset within the function (from which the line number preceding it was derived).  (I'm not even sure that that's the case; if not, just take this as a request for the offset information.)  However, it's relative to the function, which makes it really hard to match up to the assembly.  If the raw dump also contained the offset relative to the library or the absolute address, then I could combine the library base address information (in the top part of the "Raw Dump") with the binary for the release (which I can download and run through objdump -Cd) to find the instruction on which we crashed.

However, since this conveniently converted the offset to being function-relative, I have to know where the function is within the library, and without debugging symbols, that's really hard to find.


(Alternatively, Socorro could provide UI to show the assembly in context for each of the stack frames.  Even just the top one would solve most use cases, although sometimes lower frames are useful too if it's necessary to distinguish two calls that are on the same line.  That would be even better, but I'm sure it's a lot more work.)

Comment 1

[(not currently active) Ted Mielczarek]

The offset in that case is relative to the line, which is probably even less useful to you.

This is sort of hard to fix, since the raw dump is exactly that--the raw output of minidump_stackwalk. I have a half-baked project sitting around somewhere that uses SWIG to provide a Python interface to the Breakpad processor library. Ideally I'd finish that, and the Socorro processor could use that to gather more info from the minidump instead of reading the data from minidump_stackwalk's stdout.

Comment 2

[(not currently active) Ted Mielczarek]

Actually, the meaning of the instruction value depends on what info is available. You can sort of see that here, in the minidump_stackwalk source:
http://code.google.com/p/google-breakpad/source/browse/trunk/src/processor/minidump_stackwalk.cc#209
If the instruction pointer isn't in a known module, then it just gives the raw ip value.
If the instruction pointer is in a known module, but the module doesn't have symbols, then it's converted to a module-relative address.
If it's in a known module and known function, but we don't have source line data, then it's converted to a function-relative address.
If it's in a known module and function, and we have source line data, then it's converted to a source-line-relative address.

Related, I wrote up a document on manually parsing the Breakpad symbol files (which are all available via HTTP thanks to the symbol server):
https://wiki.mozilla.org/Manually_Parsing_Breakpad_Symbol_Files

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to comment #2)
> If it's in a known module and function, and we have source line data, then it's
> converted to a source-line-relative address.

... which loses useful information when the code for the source line is split into multiple pieces, no (as for bug 466024)?

Comment 4

[(not currently active) Ted Mielczarek]

Yeah. That's unfortunate. :-/

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Sounds like this is really a breakpad bug.

Comment 6

[(not currently active) Ted Mielczarek]

jimb filed an upstream Breakpad issue for this:
http://code.google.com/p/google-breakpad/issues/detail?id=340

Comment 7

[(not currently active) Ted Mielczarek]

So we rolled bug 573100 out to Socorro production, which means that if we change the Socorro frontend to expose the raw JSON instead of the pipe dump you can get this information.

Comment 8

[(not currently active) Ted Mielczarek]

This was fixed by bug 636868. In the "Raw Dump" tab nowadays, for each frame you'll find "module_offset", which is what you want. (There's also "offset", which is the instruction pointer, and "function_offset", which is the offset from the start of the function.)