417601 - Breakpad and Mac OS X completely disagree on the crashing thread and the thread contents

Status: RESOLVED INVALID

(Toolkit :: Crash Reporting, defect)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Mac OS X crash report

WARNING: URL in URL field will crash your trunk build.

While looking at bug 417485 for cl, I crashed Minefield (as I expected).

I got a nice Mac OS X crash log, which reports Thread 1 is the crashing thread (which matches cl's Camino log, to an extent).  I also submitted a mozCrashReporter report, bp-3bb35242-db48-11dc-8a7c-001a4bd43ed6

However, crash-stats is showing Thread 0 as the crasher.

In addition, crash-stats shows Thread 1 looking quite different from Mac OS X's Thread 1 (not just the issue that we're getting "nearest unstripped symbol", but the whole sequence of frames is entirely different in length.)

Filing major because if mCR is getting the wrong crashing thread (and comparing that to cl's Camino crash log, that seems without a doubt to be happening), that's a Really Bad Thing™.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Looking more closely, there seem to be two things going on here:

1) The Mac OS X crash log is picking up on the breakpad exception handler thread and inserting it between the "real" thread 0 and "real" thread 1.  I've never seen that happen before/with other crashes, but Mac-Thread 2 and crash-stats thread 1 look the same.

1a) The Mac OS X crash log might be getting the right crashing thread but then end up being off once it's inserted the breakpad exception handler thread?

2) I still think crash-stats/breakpad is getting the wrong crashing thread, based on cl's Camino stack (although the Talkback exception handler does show up there as a thread 0), but I'm rebuilding my trunk build as we speak to pick up the new crashy code, and I'll see what thread Mac OS X reports crashes it (and the thread's contents).

Er, and yeah, blocking1.9? is good; trying to work on too many bug reports at once ;)

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #1)
> up there as a thread 0), but I'm rebuilding my trunk build as we speak to pick
> up the new crashy code, and I'll see what thread Mac OS X reports crashes it
> (and the thread's contents).

Well, that's fun; my debug build won't crash on either URL from bug 417485 :/

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: breakpad 1, ardissone 0 (camino-dbg crash log)

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Since we determined bug 417485 had been fixed by today's nightly, I did a pull-by-date to match the official build Chris was using and crashed, without the benefit of exception handlers (breakpad or Talkback) getting in the way....

Comparing the the exception-handler-less stack, breakpad's pulling the correct thread; yay!

OTOH, I might have been half-right in comment 1 point 1a; there definitely seems to be an "off-by-1" type error with this crash when we get to the Mac OS X Crash Reporter.

Since breakpad's getting the right thread, dropping the severity.  I don't ever recall seeing this in any other case, so it may just be that one particular crash that was causing this problem, which would be even better.  

Ultimately I'd kind of like to know what's causing this, but it's probably not blocking1.9 material if breakpad's always getting the correct thread and crashes causing "off-by-1" are very rare.  Concur?

Sorry for the false 3-alarm here :(

Comment 5

[(not currently active) Ted Mielczarek]

FWIW, Breakpad does deliberately ignore its exception handler thread, since that thread is live when the minidump is being written, so it can't reliably gather info about it.  (And it's not really useful to crash reports anyway.)  Breakpad records the thread IDs of each thread, so assigning different ordinal numbers to them shouldn't cause any real problems.

Comment 6

[(not currently active) Ted Mielczarek]

I think this is invalid per comment 5.

Comment 7

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

The off-by-one thread disagreement between the two reporters ended up being filed upstream today as http://code.google.com/p/google-breakpad/issues/detail?id=350