772432 - New Flash related minidumps are not supported by minidump_stackwalk

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: fbf3f664-7ad7-4a8d-ad18-04957a435243.dmp

bug 769048 introduced the ability to get crash reports from Flash 262 and later. I've been seeing a large number of crashes in automation which produce minidumps which are not parseable by minidump_stackwalk.

An example is http://www.youtube.com/watch?v=_Kjv9H6BxpI&feature=related 

The attached minidump is considered invalid by vc2010 though others I produced earlier were able to be opened by it. I'll try to create more to see if I can get dumps which vc2010 consider to be valid but which are not parseable by minidump_stackwalk.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: minidump openable by vc2010 but not minidump_stackwalk

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: 17f3daca-eca0-425a-889f-58482dc6d3c6.extra

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: minidump errors

Comment 4

[Benjamin Smedberg]

I'm definitely getting *some* reasonable results from crash-stats, e.g. everything in this list:

https://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&platform=windows&range_value=1&range_unit=weeks&date=07%2F10%2F2012+13%3A25%3A18&query_search=signature&query_type=contains&query=flashplayerplugin_&reason=&build_id=&process_type=any&hang_type=any&do_query=1

It's possible that there are others which are not getting completely processed. Ted or Lars, do you know what signature a broken minidump of the sort bc is seeing would show up as in crash-stats, or whether we can poke at the processor logs to see if there are new broken minidumps of this sort?

Comment 5

[(not currently active) Ted Mielczarek]

Attachment: Breakpad patch

Rebuilding minidump_stackwalk with this patch makes it able to process these dumps. It's missing stack memory for some threads, but the main thread is okay.

Comment 6

[(not currently active) Ted Mielczarek]

Oh, only the second dump. The first one is pretty much useless, it's missing pretty much everything.

Comment 7

[Bob Clary [:bc] (inactive)]

(In reply to Ted Mielczarek [:ted] from comment #5)
> Created attachment 640671 [details] [diff] [review]
> Breakpad patch
> 
> Rebuilding minidump_stackwalk with this patch makes it able to process these
> dumps. It's missing stack memory for some threads, but the main thread is
> okay.

confirmed minidump_stackwalk can process it and a rebuilt exploitable also can process it. Thanks!

Comment 8

[(not currently active) Ted Mielczarek]

I tossed the patch upstream, we'll see how it goes:
https://breakpad.appspot.com/413002/

Comment 9

[Bob Clary [:bc] (inactive)]

One thing is it does not flag the crashing thread as crashed in the same way as other dumps. An example

...
Thread 0
...

instead of 

...
Thread 4 (crashed)
...

Comment 10

[(not currently active) Ted Mielczarek]

The problem with these dumps is that they're missing some information, or have broken information, which is why minidump_stackwalk would choke on them. My patch forces it to be more lenient, but it's still missing some info. In this case, it's probably missing the info about which thread is crashing.

Comment 11

[Bob Clary [:bc] (inactive)]

Attachment: dmp that aborts with std::__throw_out_of_range

I'm seeing several dumps that crash minidump_stackwalk with this patch. They don't crash exploitable though.

windows/cygwin version says:

terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check

the mac os x version (which you need to remove the vla error stuff) gives:

#7  0x00007fff905a61b4 in std::__throw_out_of_range ()
#8  0x000000010000451f in std::vector<google_breakpad::CallStack*, std::allocator<google_breakpad::CallStack*> >::_M_range_check (this=<value temporarily unavailable, due to optimizations>, __n=<value temporarily unavailable, due to optimizations>) at stl_vector.h:501
#9  0x0000000100004538 in std::vector<google_breakpad::CallStack*, std::allocator<google_breakpad::CallStack*> >::at (this=0x7fff5fbff650, __n=0) at stl_vector.h:537
#10 0x0000000100003006 in PrintProcessState (process_state=@0x7fff5fbff610) at minidump_stackwalk.cc:420
#11 0x00000001000032d5 in PrintMinidumpProcess (minidump_file=@0x6, symbol_paths=@0x7fff5fbff748, machine_readable=false) at minidump_stackwalk.cc:536
#12 0x0000000100001808 in main (argc=1606416256, argv=0x7fff5fbff7d8) at minidump_stackwalk.cc:586

Comment 12

[(not currently active) Ted Mielczarek]

I have an updated patch at https://breakpad.appspot.com/413002/

(It doesn't crash with the dump from comment 11, FWIW.)

Comment 13

[(not currently active) Ted Mielczarek]

This landed upstream a month ago, I forgot we had a bug open here:
http://code.google.com/p/google-breakpad/source/detail?r=1077