778162 - browser-side plugin hangs getting wrong signatures in Firefox 16+

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Robert Kaiser]

Since some point in Firefox 16.0a1 trunk, we're getting wrong browser-side plugin hang signatures again.

Example reports are bp-75717ff9-8525-4c90-8b7c-aa7852120727 (16.0a2) and bp-397498a1-fdd5-4f89-a0e4-0461a2120727 (17.0a1)

I know that it's not high priority as we're mostly ignoring the browser sides, but when we need to look into pairs, it would be nice to have useful signatures again in the future.

Comment 1

[Robert Kaiser]

So, it looks like we had the rare occurrence of 1 signature per day before 2012-06-06, but on that day, we had 6 and rose in the following days to ~60 per day on the Firefox Nightly channel.

See https://crash-analysis.mozilla.com/rkaiser/2012-06-07/2012-06-07.firefox.nightly.explosiveness.html and https://crash-analysis.mozilla.com/rkaiser/2012-06-14/2012-06-14.firefox.nightly.explosiveness.html for that stage.
This seems to match pretty much when we switched trunk from 15.0a1 to 16.0a1 - the Flash 11.3 release was on 2012-06-08, IIRC, so doesn't match as good as the version switch. Also, 15 Aurora and Beta is fine, the regression is only in 16+.

Then, on 2012-06-21, we jumped to 131, then stayed roughly between 100 and 150 for a while.

See https://crash-analysis.mozilla.com/rkaiser/2012-06-21/2012-06-21.firefox.nightly.explosiveness.html and https://crash-analysis.mozilla.com/rkaiser/2012-07-01/2012-07-01.firefox.nightly.explosiveness.html for this.
Not sure what landed on 6/20 that could have caused that. I guess what happened there (and that would match the numbers I know) is that from there, all browser-side Flash hang reports come out with this signature.

Again, on 2012-07-13, another rise starts, so that we're around 200 now. That seems to match the Flash .265 release which we know has increased hangs.
I'm not pointing to explosiveness reports for that as we know the fact that .265 increased hangs, so this additional increase is almost not report-worthy.

Comment 2

[Robert Kaiser]

And looking at https://crash-analysis.mozilla.com/bsmedberg/flash-hang-summary-vista+.svg (this is from releases, but shows overall hang counts of different Flash versions), I guess the second increase is the Flash .262 release.

So, the actual regression on our side was probably between the uplift of 15 to Aurora and the 2012-06-06 Nightly build.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

I just hit this stacktrace by having http://nbcolympics.com/ open and streaming in the background; here is the pair of incidents:

https://crash-stats.mozilla.com/report/index/bp-fff2c2ab-8cd2-4dcc-9cba-4c4902120731 - Firefox 17.0a1 Crash Report [@ hang | WaitForMultipleObjectsEx | RealMsgWaitForMultipleObjectsEx ]
https://crash-stats.mozilla.com/report/index/bp-ba79355f-e853-4b4e-82b8-cb4bb2120731 - Firefox 17.0a1 Crash Report [@ hang | WaitForSingleObjectEx | WaitForSingleObject | google_breakpad::ExceptionHandler::WriteMinidumpOnHandlerThread(_EXCEPTION_POINTERS*, MDRawAssertionInfo*) ] 

File: NPSWF32_11_3_300_268.dll
Version: 11.3.300.268
Shockwave Flash 11.3 r300

My build ID is: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0

Let me know if I should file a separate bug!

Comment 4

[Robert Kaiser]

(In reply to Stephen Donner [:stephend] from comment #3)
> Let me know if I should file a separate bug!

Yes, please. For one thing, please ignore the browser-side signature completely. For the other, please file on against Flash (we have a component in the Plugins product) with the plugin-side signature and hopefully STR.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Robert Kaiser (:[e-mail]) from comment #4)
> (In reply to Stephen Donner [:stephend] from comment #3)
> > Let me know if I should file a separate bug!
> 
> Yes, please. For one thing, please ignore the browser-side signature
> completely. For the other, please file on against Flash (we have a component
> in the Plugins product) with the plugin-side signature and hopefully STR.

Filed bug 779657.

Comment 6

[Benjamin Smedberg]

https://github.com/mozilla/socorro/pull/783 r?lars

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/4755b4452764d24a2669ec743c36990ad7d70f6d
Bug 778162 - The change from nsILocalFile to nsIFile altered the signature of CrashReporter::CreatePairedMinidumps in a way the processor wasn't aware of. Make the sentinel function more lenient in general.

https://github.com/mozilla/socorro/commit/2c5561a81cfac8e70e69201d9054ddea7db445ea
Bug 778162 - clearer way to search for a symbol in a stack, suggested by lars

https://github.com/mozilla/socorro/commit/0cdc955f81db0bd48e4f6b6a1fbe7c7d50049b5a
Merge pull request #783 from bsmedberg/parenthangsignatures

Bug 778162 - The change from nsILocalFile to nsIFile altered the CrashReporter::CreatePairedMinidumps sentinel for brower hangs. r?lars

Comment 8

[K Lars Lohn [:lars] [:klohn]]

I strongly suspect that the change in Socorro PR 783 is insufficient to correct the signature generation problem.  Using crash ba79355f-e853-4b4e-82b8-cb4bb2120731 as the canonical example, the modified signature sentinel rule accounts for the lack of parameters in the "CreatePairedMinidumps" call, but it does not account for the lack of parameters in the "mozilla::ipc::RPCChannel::Call".  Because the latter does not match anything in the call stack, the signature sentinel rule fails and is not applied by Socorro.

If I've interpreted the intent correctly, the solution is to add a second signature sentinel rule that uses literally "mozilla::ipc::RPCChannel::Call" as the sentinel.  Alternatively, the Socorro rule system could be altered to allow a regular expression as a signature sentinel instead of a literal.  With that change, both cases could be covered by one rule.

Comment 9

[Benjamin Smedberg]

I don't understand comment 8: in no case are the parameters missing, they just changed type. Perhaps you're looking at the "Details" tab of https://crash-stats.mozilla.com/report/index/ba79355f-e853-4b4e-82b8-cb4bb2120731, where the parameters are hidden by default? You can hover over the short frame to see the full signature.

Reports such as https://crash-stats.mozilla.com/report/index/34c0b15a-22d3-4e2c-829a-d29c12120812 show that this is mostly fixed. There is still a stackwalking issue with reports such as https://crash-stats.mozilla.com/report/index/a9dfa874-754a-461d-97e5-1c44e2120812, but I'm going to call that a separate bug.

Comment 10

[K Lars Lohn [:lars] [:klohn]]

ah, of course.  Disregard the ranting of the lunatic that seeks trouble where none actually exists.