805021 - [adbe 3352449] Extremely sharp spike in Flash crashes on 2012-10-23

Status: RESOLVED WORKSFORME

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Description

[Robert Kaiser]

We have an extremely large spike of Flash crashes in yesterday's data, as https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html shows we have double the crashes than the day before!

I'm not filing separate bugs for all the signatures right now, I added the main ones to this bug.

Here's a few links to more crash reports:
https://crash-stats.mozilla.com/report/list?signature=F1131028722________________________________
https://crash-stats.mozilla.com/report/list?signature=F_986496000_____________________
https://crash-stats.mozilla.com/report/list?signature=F_1407122029_____________________________________________
https://crash-stats.mozilla.com/report/list?signature=F_693444022__________________________________________________
https://crash-stats.mozilla.com/report/list?signature=F_130753751__________________________________________________________
https://crash-stats.mozilla.com/report/list?signature=F_1229035888_________________________________________________
https://crash-stats.mozilla.com/report/list?signature=F_1369831953______________________
https://crash-stats.mozilla.com/report/list?signature=F_855855466_____________________

All those jumped into the top 50 signatures overall for 16.0.1 for that single day. They were residual signatures before, but jumped extremely now.

A lot of the URLs are Farmville 2 - starting with http://apps.facebook.com/farmville-two/ - but there are YouTube and a number of others in there, esp. games on Facebook and other sites.

This isn't on a new Firefox version and not a new Flash version, it mainly happens 11.4.402.287, but also the current 11.5 beta and other 11.4 versions, even 11.3 - I expect that some change on some websites triggered that. Maybe something Zynga did? (We only report the URL of the currently active tab, so Farmville 2 could still be open in a background tab and trigger this.)

Comment 2

[smadayag]

our team is investigating.  our internal bug #3352449 is tracking the issue...

Comment 3

[Scoobidiver (away)]

Today, the Flash crash volume is three times the volume of two days ago (yesterday, it was *only* twice).
Windows XP is affected almost the same way as other Windows versions so it's not related to the Firefox Protected Mode of Flash.

Comment 4

[Robert Kaiser]

Just FYI, there's even a few more signatures involved in the spike, I only linked the highest-ranked ones - see more spiking signatures at the top of this list: https://crash-analysis.mozilla.com/rkaiser/2012-10-24/2012-10-24.firefox.16.explosiveness.html

Comment 5

[Jeromie Clark]

At this time, we believe the huge uptick here is related to the Farmville 2 crash we're seeing with 805021/3352449.  

In our investigation, we're seeing that the recorded stack varies between crashes, and I've recorded several of these signatures while reproducing this crash.

We're actively pursuing a fix for the issue, and will post an update when the next beta with a fix is available.

Comment 6

[Marcia Knous [:marcia]]

Added [@ F_986496000_____________________ ]  since it has moved into the top 30 in the most recent beta.

Comment 7

[Scoobidiver (away)]

(In reply to Jeromie Clark from comment #5)
> We're actively pursuing a fix for the issue, and will post an update when
> the next beta with a fix is available.
The fix, if it works, should be uplifted to Flash 11.4.

Comment 8

[Robert Kaiser]

The amount of Flash crashes in yesterday's data has receded somewhat, it's down to almost the level of two days before (which was still double the usual level).
I guess Zynga has reacted to the problem.

Comment 9

[chris nuuja]

Did Mozilla perhaps push a Firefox update recently?

We can reliably reproduce by turning on private browsing mode and navigating to Farmville2 on Win7 FF16.   It does not reproduce with private browsing mode on IE or incognito mode in Chrome.

Zynga is not aware of pushing anything out of the ordinary recently, they are investigating if its perhaps a new advertisement swf.

Comment 10

[Scoobidiver (away)]

(In reply to chris nuuja from comment #9)
> Did Mozilla perhaps push a Firefox update recently?
It happened the same day across all versions: https://crash-analysis.mozilla.com/bsmedberg/flash-summary.html
The latest Release Firefox update date back from October 11 well before this spike: http://www.mozilla.org/firefox/16.0.1/releasenotes/

Comment 11

[Benjamin Smedberg]

Is this spike windows-only?

Comment 12

[Robert Kaiser]

(In reply to chris nuuja from comment #9)
> Did Mozilla perhaps push a Firefox update recently?

At least not when this spike started - and it happens on multiple, even older, Firefox versions and on multiple Flash versions.

> We can reliably reproduce by turning on private browsing mode and navigating
> to Farmville2 on Win7 FF16.   It does not reproduce with private browsing
> mode on IE or incognito mode in Chrome.

Interesting. And good that we have STR. :)

> Zynga is not aware of pushing anything out of the ordinary recently, they
> are investigating if its perhaps a new advertisement swf.

Interesting. But if you have STR, the cause can potentially tracked down from there, right?

Comment 13

[Jeromie Clark]

We have identified the root-cause of the issue, and have taken short-term steps to address the crashes.  This change is server-side, and does not require a player update.

You should see a significant drop-off in incidence over the next 24 hours.  

Our next public release of the player will include a client-side fix to ensure that this does not happen in the future.

Thanks again for bringing this to our attention.

Comment 14

[Alex Keybl [:akeybl]]

(In reply to Jeromie Clark from comment #13)
> We have identified the root-cause of the issue, and have taken short-term
> steps to address the crashes.  This change is server-side, and does not
> require a player update.
> 
> You should see a significant drop-off in incidence over the next 24 hours.  

\o/ huzzah

Comment 15

[Robert Kaiser]

Benjamin, why have you resolved this bug as WFM? This hasn't come down to even nearly previous levels yet, as far as I can see.

Comment 16

[chris nuuja]

At what time do you collect stats for crashes and what time period does that collection cover?    The server side change was pushed late yesterday and it may take some time for it to percolate through all the servers.  By late last night all the server changes should have been active.

Comment 17

[Alex Keybl [:akeybl]]

(In reply to chris nuuja from comment #16)
> At what time do you collect stats for crashes and what time period does that
> collection cover?    The server side change was pushed late yesterday and it
> may take some time for it to percolate through all the servers.  By late
> last night all the server changes should have been active.

If this was rolled out late last night, we should have a clearer picture of whether or not this crash is dropping significantly tomorrow morning.

Comment 18

[Robert Kaiser]

FYI, we're collecting reports continously, of course, and it's usually possible to query them with one or two hours of delay. All our stats are usually done for full UTC days, though, so we usually only see significant changes the day after they have happened, for fixes it often takes a day or two more to see numbers stabilize back to the expected level.

Comment 19

[Robert Kaiser]

That said, it really looks like numbers for yesterday are back to where they were before this spike, see the first link I gave in comment #0.