616358 - tracking bug for sharp spike in flash crashes around nov 3.

Status: RESOLVED WORKSFORME

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Description

[chris hofmann]

there is some unusual crash spike data for flash crashes starting around nov3.

seems to be showing up uniformly across several flash signatures (at least the top 3 looked at so far) and possibly across firefox versions.

more preliminary data at

#1 top flash crash
F957328252________________________
https://bugzilla.mozilla.org/show_bug.cgi?id=585913#c9

#2 top flash crash
F_20680835_________________________________
https://bugzilla.mozilla.org/show_bug.cgi?id=586903#c5

#3 top flash crash
F1146177332__________________________
no bug on file yet, but some data for this is listed below

20101101 66  	33 3.6.122010102621, 
        		7 4.0b62010091408, 	5 3.6.82010072215, 
        		5 3.6.102010091412, 	4 3.6.32010040108, 
        		3 3.6.112010101211, 	2 4.0b8pre2010102504, 
        		2 3.5.152010102620, 	1 4.0b7pre2010100204, 
        		1 3.6.22010031607, 	1 3.62010011514, 
        		1 3.5.142010100117, 	1 3.1b32009030423, 
20101102 37  	19 3.6.122010102621, 
        		5 3.6.102010091412, 	4 3.6.82010072215, 
        		3 4.0b62010091408, 	3 3.5.152010102620, 
        		1 3.6.32010040108, 	1 3.6.112010101211, 
        		1 3.62010011514, 
20101103 302  	196 3.6.122010102621, 
        		23 4.0b62010091408, 	12 3.6.32010040108, 
        		12 3.6.102010091412, 	11 3.6.82010072215, 
        		9 3.5.152010102620, 	6 3.6.112010101211, 
        		5 3.62010011514, 	4 3.0.192010031422, 
        		3 3.6.62010062523, 	2 4.0b8pre2010110204, 
        		2 4.0b42010081813, 	2 3.5.92010031508, 
        		1 4.0b8pre2010110104, 	1 4.0b8pre2010103104, 
        		1 4.0b8pre2010101404, 	1 4.0b22010072019, 
        		1 4.0b12010063014, 	1 3.6.92010082415, 
        		1 3.6.42010061114, 	1 3.6.22010031607, 
        		1 3.5.82010020216, 	1 3.5.32009082410, 
        		1 3.5.142010100117, 	1 3.52009062402, 
        		1 3.0.62009011913, 	1 3.0.152009101601, 
        		1 3.0.12008070208, 
20101104 382  	269 3.6.122010102621, 
        		38 4.0b62010091408, 	14 3.6.102010091412, 
        		12 3.6.32010040108, 	10 3.6.112010101211, 
        		7 3.6.82010072215, 	7 3.5.152010102620, 
        		5 3.62010011514, 	3 3.6.62010062523, 
        		2 3.6.42010061114, 	2 3.6.22010031607, 
        		2 3.5.142010100117, 	2 3.0.192010031422, 
        		1 4.0b8pre2010110304, 	1 4.0b7pre2010100604, 
        		1 3.6b52009120414, 	1 3.6.92010082415, 
        		1 3.5.22009072922, 	1 3.5.112010070102, 
        		1 3.5.102010050409, 	1 3.0b52008032620, 
        		1 3.0.62009011913,

Charles, are you guys aware with any conditions that might be related to this?

Folks running firefox 4.0 latest development builds have been noticing a sharp uptick in recent flash crashes, but this looks uniform across all releases.  It might be masking other problems with recent landings in firefox 4 development builds.

We also need to figure out if this could be a data collection or processing problem contributing to the higher numbers.  for some reason these signatures are showing up marked as browser crashes. See: https://bugzilla.mozilla.org/show_bug.cgi?id=615961

not sure how to set up the dependency for this tracking bug, but I took a stab at marking 58591,586903,615961 as depends on this bug, since reducing the numbers on those crashes might depend on figuring out whats going on here.

Comment 1

[chris hofmann]

so the links to the other bugs and that data above shows about a 2x, 3x, and 6x increase in the top 3 flash crashes between nov2 and nov3.

Comment 2

[chris hofmann]

overall it looks like the total for flash crashes jumped from 19-20,000 to 33-36,000 in the days before and after nov3.

20101101 20431
20101102 19988

20101103 33623
20101104 36918
20101105 36090

Comment 3

[chris hofmann]

Attachment: comparision of top 50 flash crashes on nov. 2 and 3

Comment 4

[chris hofmann]

Attachment: compare mac flash crashes on nov 2 and 3

if we look at just the mac crashes for those two days we don't see the same spike.

Comment 5

[chris hofmann]

Attachment: compare mac crashes nov2 and dec1

crashes on mac look like they have declined when comparing early nov. and early dec.

Comment 6

[Scoobidiver (away)]

How long have crashes been kept in Socorro?
These spikes took place exactly one month ago.

Comment 7

[chris hofmann]

we are keeping data forever right now.

Comment 8

[chris hofmann]

There is at aleast one mac crash that seems to have appeared just after the big windows spike.   seems to affect smooney a lot.

bug 616471

Comment 9

[chris hofmann]

http://www.adobe.com/support/security/bulletins/apsb10-26.html  indicates a flash update on Nov. 4.    That matches with the introduction of several new Mac signatures, but the windows spike seems to have happened a day or two before.  cliss, is is possible the windows roll out started happening for some users on nov. 3rd and these spikes are connected to the latest flash release?

Comment 10

[Charles]

Thanks Chris.  There was no Flash Player release at this time so its possible we are witnessing some content changes.  We'll take a look see what we can find. Might be worth scanning for URLs to see if there is a story there.

Comment 11

[smadayag]

this may also correspond to the OS X 10.6.5 release.  read that it was originally planned to ship early november. how best to strip the crash information between 10.6.4 vs 10.6.5?

Comment 12

[chris hofmann]

we don't record flash version info for mac, so that would be hard.

I notice in this report that FlashPlayer-10.6 and Flash Player are loaded.  Would that cause prblems?   click on the "modules" tab and scan down the page.

http://crash-stats.mozilla.com/report/index/04caf516-41db-4b39-bd72-e78642101127

Comment 13

[chris hofmann]

on windows we see a pretty strong correlation to a crash volume increase on the topcrash signature with the lastest flash release:  see  https://bugzilla.mozilla.org/show_bug.cgi?id=585913#c10

Comment 14

[Jet Villegas (inactive)]

Picking this up.

Comment 15

[PaulV]

If it helps to diagnose this, I started getting these crashes immediately after upgrading OSX to 10.6.5, which shipped a new Flash (http://support.apple.com/kb/HT4435).

    File: Flash Player.plugin
    Version: 10.1.102.64
    Shockwave Flash 10.1 r102

There seems to be rendering issue (e.g. on happytc.com, if I switch tabs/windows, the flash disappears and can sometimes be coaxed to reappear piecemeal by waving the mouse over it and clicking) which may be linked to the crashes.

I can't reproduce on Safari 5.0.3 (6533.19.4), so it's not just a case of a really unstable flash release.

Comment 16

[smadayag]

Thanks for pointing out your particular issue with the os/player (10.6.5/10,1,102,64) upgrade.  we'll have to sift through the dumps to confirm the other reports.

Comment 17

[Marcia Knous [:marcia]]

Hello smadayag - Have you had any luck sifting through the dumps?

(In reply to comment #16)
> Thanks for pointing out your particular issue with the os/player
> (10.6.5/10,1,102,64) upgrade.  we'll have to sift through the dumps to confirm
> the other reports.

Comment 18

[smadayag]

Nothing conclusive. Seems the reported issue is not platform specific as post#1 is stating windows but post15 is stating mac.

Jethro is following up with dev on the issue...

Comment 19

[chris hofmann]

any updates on this?

Comment 20

[smadayag]

unfortunately, there was so much going on between OS updates to the Salt release, we couldn't pinpoint the issue.  we placed focused on the individual crash bug fixes and have seen a reduction in the reports.  F957328252 is an ongoing issue...  we took a fix for it and it has dropped from 62553hits to 10469hits the same time since 1/1/11.  it's still a high reported number and we have entered another internal bug to track any new work. in bugzilla, it is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=585913

closing this bug as works for me as the spikes from november of last year have dropped...  at least for the ones noted in this bug.