559661 - minidumps missing CodeView records for some modules (some crash stacks don't get mapped to file and line number)

Status: RESOLVED WORKSFORME

(Toolkit :: Crash Reporting, defect)

Description

[Timothy Nikkel (:tnikkel)]

Some crash stacks don't get mapped to file and line number, just having xul.dll@0x10c915. For example these three from one computer (which also produces crashes with file and line number mappings):

http://crash-stats.mozilla.com/report/index/bp-2c2e5910-cf2f-4658-a542-617892100415
http://crash-stats.mozilla.com/report/index/bp-1ef7c227-51e4-4f6e-a370-5eefb2100413
http://crash-stats.mozilla.com/report/index/bp-94286fa7-cca4-40f7-ad54-7eec92100323

Comment 1

[timeless]

timothy, can you see if:

http://developer.mozilla.org/En/How_to_get_a_stacktrace_with_WinDbg

yields a better stack trace?

Comment 2

[Timothy Nikkel (:tnikkel)]

Sure, but that doesn't really help as these were crashes that I don't have steps to reproduce and I can't run with windbg attached all the time, and users won't do that either.

Comment 3

[Arie Paap [:wildmyron]]

Most modules in those reports have blank Debug Identifiers - including xul.dll in 2 of the 3 and many OS dlls

Comment 4

[Timothy Nikkel (:tnikkel)]

Is the last comment something I need to understand and/or respond to? Cause I don't understand it. :)

Comment 5

[timeless]

timothy: not precisely, although we do need information from you.

please follow the instructions here:
http://www-archive.mozilla.org/quality/help/dependency-walker.html

I need the .dwi file, you can zip it, you don't need to run firefox for very long, just start firefox and quit.

if you can't attach the zip file, http://support.mozilla.com/chat should provide a way for you to get the file to me.

Comment 6

[Timothy Nikkel (:tnikkel)]

Attachment: dwi

Here it is.

Comment 7

[timeless]

so, the version number for all of the gecko versioned files is consistent:
c:\firefox3.7nightly\firefox\XPCOM.DLL 1.9.3.3767

from memory the '3767' is theoretically meaningful (worst case someone can hunt through the archives and identify it). i don't see anything wrong with the dll list.

tn: thanks for the file, someone else might use it too. i think we're going to need ted to comment.

wildmyron: i think you're going to want to grab the binary and try it yourself. the file dates are 4/25/2010 1:48pm (not sure whose timezone that is, perhaps mine in which case 3:48am pacific).

Comment 8

[(not currently active) Ted Mielczarek]

(In reply to comment #7)
> so, the version number for all of the gecko versioned files is consistent:
> c:\firefox3.7nightly\firefox\XPCOM.DLL 1.9.3.3767
> 
> from memory the '3767' is theoretically meaningful (worst case someone can hunt
> through the archives and identify it). i don't see anything wrong with the dll
> list.

It's "days since January 1, 2000":
http://mxr.mozilla.org/mozilla-central/source/config/version_win.pl#254

so 3767 should be 2010-04-25.

Comment 9

[Timothy Nikkel (:tnikkel)]

Is it important that I use a specific build? Because I just used the latest nightly when I made the dwi file since I can't reproduce a crash without linenumbers when I want to.

Comment 10

[timeless]

the reason for dependency walker was to hunt for hints about old files from incorrect versions. so if you weren't using it with the unhappy crashing binary it didn't tell us anything useful.

Comment 11

[Timothy Nikkel (:tnikkel)]

The three crashes in comment 0 happened with three different builds.

Comment 12

[(not currently active) Ted Mielczarek]

I investigated this in bug 575100 comment 6 (we saw it in a different crash report). There's data missing from the module entries in the crash report. I'm not sure what the root cause is, or that we have any way to fix that directly, since we're just using DbgHelp.dll!MinidumpWriteDump, but I was able to figure out why the debugger can get the symbols. I think I might fix this as part of bug 528092, since internally it's related. I filed an upstream bug with a proposal there:
http://code.google.com/p/google-breakpad/issues/detail?id=389

Comment 13

[(not currently active) Ted Mielczarek]

I think this is just a subcase of OOM crashes producing empty dumps--I believe that MinidumpWriteDump memory-maps each DLL in order to get its debug ID, and xul.dll is kind of big, so when we run out of virtual memory sometimes dump writing succeeds but getting the debug ID fails. I think the fix from bug 837835 / bug 943051 should reduce the frequency of this sort of problem.