535071 - Linux crash report file ID doesn't match file ID from dumped symbols

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Daniel Holbert [:dholbert]]

My crash reports from today's nightlies haven't had useful stack information -- instead of function names, they just have module name + offset.

For example: I just triggered the crash from bug 535004, using a nightly from yesterday and a nightly from today.  Here are the results:

Yesterday's nightly (20091214030830):
http://crash-stats.mozilla.com/report/index/801b2e3e-e575-4240-8c07-716822091215
* Title has signature "[@ nsSMILCSSProperty::GetBaseValue() const ]"
* Stack has symbol information (names of functions)

Today's nightly (20091215030948):
http://crash-stats.mozilla.com/report/index/94e9fb23-5eb6-416d-a58e-729162091215
* Title has signature  "[@ libxul.so@0x7fa124 ]"
* Stack has no symbol information -- just has e.g. "libxul.so@0x7f9c3e" at each stack level.

Comment 1

[Daniel Holbert [:dholbert]]

Pushlog for regression range:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=44c392db6672&tochange=96e8d529b2d3

Just got a new crash report from each build again, with the same results.

Yesterday's nightly (with symbols):
http://crash-stats.mozilla.com/report/index/960fd911-0d3b-4cdd-aad4-6706f2091215
Today's nightly (no symbols):
http://crash-stats.mozilla.com/report/index/0a8bee82-04b9-4cb5-957c-9d9422091215

The changes for bug 514188 ("sync to breakpad revision 437 to pick up Linux client rewrite") sounds particularly suspicious...  I guess the out-of-process-plugins patch-bomb seems suspicious too, if only because of its size.  I'm blaming bug 514188 for now, though...

Comment 2

[(not currently active) Ted Mielczarek]

There's something...odd about that second crash report. We have symbols for the build with that build ID:
http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20091215030948-symbols.txt

However, the libxul.so there has the ID:
libxul.so/6429391C3C6500D5A2B4C500E40E1A000/libxul.so.sym

but the libxul.so from your crash report has the ID:
468C329B3C6500D5A2B4C500E40E1A000

Those don't match. So either you're running a mixed-up build, or something is broken between the symbol dumping and crashing bits.

Comment 3

[(not currently active) Ted Mielczarek]

Jim: don't suppose any of your symbol dumper changes could have caused something weird to happen here?

Comment 4

[(not currently active) Ted Mielczarek]

Ugh. The Linux client rewrite definitely broke this. Apparently the Linux File ID code changed to just calculate a hash of the first page of the file, which changes if you strip the file or anything. I guess I'll back out until I can figure out how to fix it.

Comment 5

[(not currently active) Ted Mielczarek]

I filed an upstream issue on this brokenness:
http://code.google.com/p/google-breakpad/issues/detail?id=357

Comment 6

[(not currently active) Ted Mielczarek]

Backed out, bug 514188 reopened. Apparently bsmedberg is going to get some new nightlies spun today to fix some e10s stuff, so the new nightly today should be better.

I'll try to get this sorted out upstream so we can re-land, since the new code is otherwise much better.

Comment 7

[Phil Ringnalda (:philor)]

Was it expected that your backout would actually *cause* this, for Linux talos crashes? Prior to your backout, and for one of the two that tbpl claims were from your backout, the bug 528708 crashes all had the libgtk-x11-2.0.so.0.1200.0 signature, then for one from the backout and all following ones, they're (assuming they really are the same thing) all just libxul.so + 0xddf84 (or 0xddf44, or thereabouts).

Comment 8

[(not currently active) Ted Mielczarek]

Uh, no. The backout should have just reverted us to the previously existing Breakpad code. (I forgot to look into what happened with the debug symbols on tinderbox, I would expect things to continue working.)