1228735 - Flash plugin crash rate doubled on 43beta on windows 7 since 2015-11-23

Status: RESOLVED DUPLICATE of bug 1227221

(Firefox :: Untriaged, defect)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-07498fc5-bfc6-493e-a94c-d982f2151123.
=============================================================
[Tracking Requested - why for this release]:

something very curious is going on with the plugin crash rate on firefox beta 43 at the moment - on the crash graph at https://crash-analysis.mozilla.com/rkaiser/crash-report-tools/longtermgraph/?fxbeta it can be observed that the plugin crash rate doubled from 0.9% to 0.17% on 2015-11-23. nothing comparable happened on any other channel.

looking further into the matter it seems that this rise is solely attributable to an increase in flash crashes on windows 7 and it looks unrelated to the plugin hang in flash 20 beta that was brought up in the last channel meeting (bug 1227221).

this phenomenon can be best observed by looking at the graph for the signature npswf32_16_0_0_228.dll@0x35acbd which had a baseline of near zero before november 23rd:
https://crash-stats.mozilla.com/signature/?date=>2015-11-01&version=43.0b&signature=npswf32_16_0_0_228.dll%400x35acbd#graphs

the distribution of crashing flash plugin versions changed dramatically since then. in the week before november 23rd they looked what you'd expect them to be (capped at a 100 crashes per week):
Plugin version facet Rank 	Plugin version 	Count 	%
1 	19.0.0.245 	5617 	54.18 %
2 	20.0.0.214 	2342 	22.59 %
3 	19.0.0.226 	632 	6.10 %
4 	20.0.0.203 	387 	3.73 %
5 	19.0.0.185 	104 	1.00 %
https://crash-stats.mozilla.com/search/?product=Firefox&version=43.0b&date=%3C%3D2015-11-22&date=%3E2015-11-14&platform_pretty_version=%3DWindows+7&plugin_name=Shockwave+Flash&_facets=signature&_facets=user_comments&_facets=version&_facets=plugin_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-plugin_version

...whereas since then they have changed to include loads of very outdated versions:
Rank 	Plugin version 	Count 	%
*1 	19.0.0.245 	5741 	29.84 %
2 	16.0.0.228 	3055 	15.88 %
*3 	20.0.0.214 	2006 	10.43 %
4 	16.0.0.287 	1067 	5.55 %
5 	18.0.0.107 	1051 	5.46 %
*6 	19.0.0.226 	808 	4.20 %
7 	18.0.0.232 	676 	3.51 %
8 	20.0.0.231 	598 	3.11 %
9 	18.0.0.209 	563 	2.93 %
*10 	19.0.0.185 	537 	2.79 %
11 	19.0.0.207 	277 	1.44 %
12 	16.0.0.305 	273 	1.42 %
13 	17.0.0.169 	226 	1.17 %
14 	17.0.0.188 	219 	1.14 %
15 	18.0.0.129 	171 	0.89 %
16 	20.0.0.229 	149 	0.77 %
17 	19.0.0.162 	139 	0.72 %
18 	17.0.0.134 	129 	0.67 %
19 	18.0.0.194 	124 	0.64 %
20 	19.0.0.219 	118 	0.61 %
https://crash-stats.mozilla.com/search/?product=Firefox&platform_pretty_version=%3DWindows+7&version=43.0b&plugin_name=Shockwave+Flash&date=%3E%3D2015-11-23&_facets=signature&_facets=user_comments&_facets=version&_facets=plugin_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-plugin_version

i'm at a loss to explain what's going on there. the most affected beta 43.0b4 was already released for nearly a week before this huge rise happened on november 23 and i know of no other events that day that could influence the plugin crash rate that noticably and only on beta...

Comment 1

[(No longer employed by Mozilla) Aaron Klotz]

Jeromie, do you have any insights on your end as to why this might be happening? Perhaps a deobfuscated crash stack might be of use here...

Comment 2

[Jeromie Clark]

Yeah, we've in touch with Mozilla at an engineering level since last week about this.  See 1227221 for more details.

While we're still investigating, at this point, we believe that this is related to the Tab Muting feature introduced in Firefox 42.  We had added APIs specifically for NPAPI Flash Player in Safari in our Quint release, but it looks like they also got picked up in Firefox; however, we didn't know about that until the issues started rolling in.

We're continuing to investigate, but that's the working theory, and we'll hopefully have that proved out today.  If that's the case, we're probably going to disable those APIs outside of Safari for the moment, and then look at explicitly adding support for Firefox in a controlled, well-tested way.

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Already tracking bug 1227221.