1227221 - [ADBE 4091133] Flash 20.0.0.214 (beta) shows a huge amount of plugin hangs

Status: RESOLVED FIXED

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Description

[Robert Kaiser]

[Tracking Requested - why for this release]:

I saw a large increase of "plugin hangs" in our stats for Firefox beta starting on Nov 20 (maybe end of Nov 19 UTC) and finally could track this back to being caused by the newest Flash beta, version 20.0.0.214.

What we call a "plugin hang" in this context is the plugin process not reacting to pings from the main browser process for a long time (IIRC 45s) and us then killing the plugin process and showing a crash reporter dialog.

We saw the amount of those hangs roughly double compared to the previous pretty constant level.

This search shows that 2/3 of all plugin hangs on our most recent beta in the last days are from this new beta version of Flash:

https://crash-stats.mozilla.com/search/?product=Firefox&version=43.0b5&process_type=plugin&plugin_name=Flash&hang_type=hang&date=%3E2015-11-20&date=%3C2015-11-23&_facets=signature&_facets=plugin_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-plugin_version

On our current release, which has way fewer Flash beta users, this is not as visible:
https://crash-stats.mozilla.com/search/?product=Firefox&version=42.0&process_type=plugin&plugin_name=Flash&hang_type=hang&date=%3E2015-11-20&date=%3C2015-11-23&_facets=signature&_facets=plugin_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-plugin_version


Looking at the rate of hangs vs. crashes, the current Flash release has them at 56% hangs:
https://crash-stats.mozilla.com/search/?product=Firefox&process_type=plugin&plugin_name=Flash&plugin_version=19.0.0.245&date=%3E2015-11-20&date=%3C2015-11-23&_facets=signature&_facets=hang_type&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-hang_type

The new beta has 80% of reported issues being hangs:
https://crash-stats.mozilla.com/search/?product=Firefox&process_type=plugin&plugin_name=Flash&plugin_version=20.0.0.214&date=%3E2015-11-20&date=%3C2015-11-23&_facets=signature&_facets=hang_type&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-hang_type

The main hang signature there is in F_1706698279, which is usually much less pronounced (the F_1152915508 ones usually dominate), but there are more different ones:
https://crash-stats.mozilla.com/search/?product=Firefox&process_type=plugin&plugin_name=Flash&plugin_version=20.0.0.214&date=%3E2015-11-20&date=%3C2015-11-23&hang_type=hang&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Comment 1

[Jeromie Clark]

Thanks.  We're looking into it.

Comment 2

[Jeromie Clark]

This is ADBE 4091133

Comment 3

[Chris Peterson [:cpeterson]]

Someone on Reddit reports Flash hangs from a recent Windows Update, but I don't know what Flash version they are running:

I can confirm that installing the optional Windows 10 application compatibility runtime (Universal CRT) in Windows Update KB2999226 precipitates the problem state where flash player for all browsers and the Firefox plugin container hang due to high CPU usage in Windows Vista x64. Removing this update resolves problems with flash and plugin container for all browsers in Windows Vista x64.

https://www.reddit.com/r/firefox/comments/3tyomz/possible_cause_of_flash_player_and_plugin/

Comment 4

[Jeromie Clark]

I think you got it, thanks.  There's a 100% correlation between this hang, flash 20.0.0.214 and vista.

https://crash-stats.mozilla.com/search/?product=Firefox&process_type=plugin&plugin_name=Flash&hang_type=hang&version=42.0&plugin_version=20.0.0.214&_facets=signature&_facets=plugin_version&_facets=platform_pretty_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-plugin_version

Comment 5

[Jeromie Clark]

Sorry, I totally read that wrong. If we're talking about the uptick on 1706698279, it's correlating 100% to WinXP.

https://crash-stats.mozilla.com/report/list?product=Firefox&range_unit=days&process_type=plugin&version=Firefox%3A43.0b5&hang_type=hang&signature=hang+%7C+RtlpWaitForCriticalSection+%7C+RtlEnterCriticalSection+%7C+F_1706698279__________________________________&date=2015-11-23+00%3A00%3A00&range_value=14

Comment 6

[Robert Kaiser]

(In reply to Jeromie Clark from comment #5)
> Sorry, I totally read that wrong. If we're talking about the uptick on
> 1706698279, it's correlating 100% to WinXP.

We're seeing the other generic hang signatures also up, which are on the other Windows versions. It looks to me like this is probably increasing hangs across the board, but usually we have a really low number of hangs on WinXP (non-protected-mode) while the protected mode hangs in F_1152915508 (multiple signatures) are most of the hangs we're seeing. With this beta, it seems, all variants are hanging and adding that to the usual "almost nothing" volume of the XP signature is way more visible than adding it to the already higher volume of the other signatures. That said, it's a good find that 1706698279 is XP. :)

Comment 7

[Jeromie Clark]

Understood, thanks.  After further investigation, we believe that these are related to a couple deadlocks that were identified and fixed in our codebase last week, but those changes haven't landed in a public beta.

We're building a new beta tonight and will push that out to the beta audience in the morning to confirm.  Please expect symbols from someone on the US team early tomorrow morning.  

We're pretty close to cutting GM, so hopefully we can establish that we're near baseline again over the next couple days, at least to the extent possible.

Thanks again for the heads-up on this.

Comment 8

[Robert Kaiser]

(In reply to Jeromie Clark from comment #7)
> We're pretty close to cutting GM, so hopefully we can establish that we're
> near baseline again over the next couple days, at least to the extent
> possible.

Thanks, we'll be watching the data, great to hear that you believe you have found the issue.

Comment 9

[[:philipp]]

it doesn't look like 20.0.0.231 has improved the situation.

Comment 11

[Jeromie Clark]

Yes, we arrived at that conclusion yesterday.  We're pretty sure that we know what's going on at this point, and I believe folks will be reaching out to Johnny later this afternoon.

Comment 12

[Liz Henry (:lizzard) (relman/hg->git project)]

Jeromie, any news here?

Comment 13

[Jeromie Clark]

Yeah, this is resolved in our daily builds.  We'll be temporarily disabling tab muting in Flash outside of Safari while we get the deadlock in Firefox resolved.  

I don't know off the top of my head whether or not we're going to push another beta before our GM goes live.  If we did, it would be sometime early next week.  I'll check with the team and find out, but this issue won't be hitting the release audience.

As usual, I'm grateful for the beta program and for the great crash telemetry coming in from Firefox.  :)

Comment 14

[Liz Henry (:lizzard) (relman/hg->git project)]

Adding tracking flag for 43.

Comment 15

[Liz Henry (:lizzard) (relman/hg->git project)]

So as I understand it, once 43 is in the release channel, this won't show up. Jorge does that sound right?

Comment 16

[Liz Henry (:lizzard) (relman/hg->git project)]

Or do we still need to block this (either on beta or release) to prevent the hangs or crashes?

Comment 17

[Jeromie Clark]

The next update to Flash Player will resolve this.  No change required in Firefox.

Comment 18

[Jorge Villalobos [:jorgev] (he/him)]

As I understand this bug, this only applies for people who have a beta version of Flash installed, and this should be fixed once the next Flash update is pushed.

We've never blocked a beta version of the plugin before. It should be possible, but it sounds like Adobe will fix this on their side and there's no action required for us.

Comment 19

[Jeromie Clark]

Yes, that is correct.  The beta is intended to surface issues like this before they make it to the release channel, so it would be surprising to see it blocked over a functional issue.  

If this is causing a huge support headache for Mozilla or there are other pain-points that are causing significant problems for Mozilla at the moment, please let me know and we'll see what we can do to expedite a remediation for impacted Flash beta users.

Comment 20

[Jeromie Clark]

FYI, we pushed Flash Player 20.0.0.238 to our beta audience today, which should have this fix.

Comment 21

[Robert Kaiser]

The hangs have gone down earlier, but crashes spiked about the same amount. That said, starting this Monday, Dec 14, we are back to normal levels again with both crashes and hangs, so I'm calling this fixed.

Comment 22

[Liz Henry (:lizzard) (relman/hg->git project)]

Marking as fixed based on comment 21.