Closed Bug 632937 Opened 13 years ago Closed 12 years ago

Need comparison of Flash 10.1 and Flash 10.2 crash rates and topcrashes (trunk and branch)

Categories

(Socorro :: Data request, task)

Product:
Socorro
Component:
Data request
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: benjamin, Assigned: laura)

References

Details

(Whiteboard: depends on 630312)

Attachments

(2 files, 3 obsolete files)

firefox3.6, crashes grouped by flash 10.1 and flash 10.2
51.64 KB, text/csv
Details
firefox4.0b11, crashes grouped by flash 10.1 and flash 10.2. This file is only for reference as the ADUs for Flash 10.x on Fx4b11 cannot be correctly determined.
217.23 KB, text/csv
Details 
I need some custom topcrash reports made available separated by Flash 10.1 and Flash 10.2. I think this is going to require some map-reduce jobs, because the Flash version is not stored anywhere else in the database.

The purpose of these reports is to take equivalent versions of Firefox (3.6.13 and 4.0b11, say) and see what crashes have been fixed in Flash 10.2, and which new ones have been introduced.

Because adoption of Flash may ramp up slowly, I'm not exactly sure how to get comparable numbers, I'd love statistical suggestions here.

On Windows, the version of Flash is reported in the module list under NPSWF32.dll.

On Mac, the version of Flash is also in the module list, but it's a bit more complicated. Flash 10.2 is listed as "FlashPlayer-10.6", doesn't have a version entry, but does have a debug ID of 91A9F4C9E35FC25E987C8EFF94EB3A440.

For older versions of Flash, the module is "Flash Player". I think that if we group the results by the debug ID, that should be sufficient.

This is a fairly high priority, I'd like preliminary results by Monday if possible.
Need to see how well aphadke can do this analysis since xstevens will be on PTO the rest of the week.
Assignee: nobody → aphadke
paraphrasing IRC conversations between benjamin and anurag:

Build a list of all signatures for Flash crashes on Firefox 3.6.13/Flash 10.1 and Firefox 3.6.13/Flash 10.2
Restrict to Windows for first iteration.

Flash version can be found by looking at:
NPSWF32.dll on frame 0 of raw_dump (or crash_signature inside HBase)
Look for corresponding flash version in the module list:
Module|NPSWF32.dll|10.2.152.26|NPSWF32.pdb|AE9732643EEF42A7B7DBF7BC85F2EDB21|0x5fcb0000|0x60296fff|0
10.2.152.26 is the flash version, we are interested in 10.2.xxx

Compare via percentage as 10.2 population might be small.
Repeat for:
Firefox 4.0b11/Flash10.1 and Firefox 4.0b11/Flash 10.2
Whiteboard: ETA: 2/10 for first iteration
flash version info is in the .csv files @ field 22

./format.sh 20110207*
1 signature
2 url
3 uuid_url
4 client_crash_date
5 date_processed
6 last_crash
7 product
8 version
9 build
10 branch
11 os_name
12 os_version
13 cpu_info
14 address
15 bug_list
16 user_comments
17 uptime_seconds
18 email
19 adu_count
20 topmost_filenames
21 addons_checked
22 flash_version
23 hangid
24 reason
25 process_type
26 app_notes

simple command to show counts of all firefox and flash version pairs for all the NPSWF32.dll signatures on 2011 02 07 looks like

 awk -F\t '$1 ~ /NPSWF32.dll/ {print $8,$22}' 20110207* | sort | uniq -c | sort -nr | more
1275 3.6.13 10.0.32.18
1174 3.6.13 10.0.42.34
1065 3.6.13 10.0.22.87
 876 3.6.13 10.0.12.36
 451 3.5.16 10.0.32.18
 285 3.6.13 10.2.161.23
 262 3.5.16 10.0.22.87
 260 3.5.16 10.0.42.34
 247 3.5.16 10.0.12.36
 188 4.0b10 10.0.22.87
 155 4.0b10 10.0.32.18
 153 4.0b10 10.0.12.36
 142 3.0.19 10.0.22.87
 140 3.6 10.0.42.34
 135 4.0b10 10.0.42.34
 121 3.6.13 10.0.2.54
 119 4.0b10 10.2.152.14
 117 3.6.13 10.2.152.14
 114 3.0.19 10.0.12.36
 113 3.6.13 10.1.102.64
 106 4.0b10 10.2.161.23
 104 3.6.13 9.0.124.0
 104 3.0.19 10.0.32.18

Data for each day is in the pub-crashdata files at places like

http://people.mozilla.org/crash_analysis/20110208/
quick look shows relatively the same numbers of flash 10.1 and 10.2 crashes for all NPSWF32.dll signatures on feb7 for 3.6.13 but 3x more for flash 10.2 on 4.0b10

   flash10.1 flash10.2
3.6.13 337 440
4.0b10  77 299
4.0b9    5  12
those numbers seem to hold up looking at a larger sample for feb 1-7

join flash10.1 flash10.2
3.6.13    2257 2829
4.0b10     487 2110
4.0b9       59  100
4.0b9pre     1    1
the regex might need a bit of refinement but if I just add $1 ~ /^F/ to pick up all the flash crashes like

11094 F957328252________________________
1048 F_1441563101_______________________
 987 F_437410032______________________________________________________________
 947 FFPlugin.dll@0x19d0c4
 666 F1146177332__________________________
 491 FlashPlayer-10.6@0x2ace83
 408 FFPlugin.dll@0x173ee4
 392 F1559406643________________________________________
 368 F_1115197261____________________________________________________________

I get different results where total flash crashes on 10.2 are only 23-25% compared to 10.1 across both firefox 3.13 and 4.b10

join   flash10.1 flash10.2
3.6.13    171929 40182   23%
4.0b10     33412  8361   25%
3.6.3       5452  3053
3.6         4546  2965
3.6.8       3238   848
4.0b9       1883   537
4.0b7       1637   474
3.6.6       1537   432
4.0b8       1461   383

there is a bit of noise in the matches for /^F/ but not much.
Are the higher numbers for overall NPSWF32.dll crashes on 10.2 and firefo 4 just a possible indication that we need some symbols for specific releases of flash and a few people in the beta population just happen to be on those versions of flash in higher relative numbers than the general public?  Looks like the concentration is on flash 10.2.161.23 and 10.2.152.14

counts  ff  flash version

 285 3.6.13 10.2.161.23
 106 4.0b10 10.2.161.23

 117 3.6.13 10.2.152.14
 119 4.0b10 10.2.152.14

 113 3.6.13 10.1.102.64
  11 4.0b10 10.1.102.64

  66 3.6.13 10.1.51.66

  44 3.6.13 10.1.51.45
  18 4.0b10 10.1.51.45

  40 3.6.13 10.1.51.95

  29 3.6.13 10.2.152.21
  68 4.0b10 10.2.152.21

  21 3.6 10.1.51.95
  10 4.0b10 10.1.51.95

  18 3.6.13 10.1.52.14
  26 4.0b12pre 10.2.152.14

  16 3.6.13 10.1.53.7
  12 3.6.13 10.1.53.64
  10 3.6.13 10.1.85.3

  10 3.6.12    10.2.161.23
  11 4.0b12pre 10.2.161.23
  10 4.0b9     10.2.161.23
Considering that Flash 10.2 final was not released until Tuesday afternoon, I really don't think numbers before 8-Feb are going to be that interesting. I'm interested in all the plugin crashes with the Flash plugin-container, not just the /^F/ crashes.

The hard part of this is figuring out what part of the population has Flash 10.1 and Flash 10.2: you really can't compare just the number of crashes, or we won't know whether the total volume has gone up significantly. Flash adoption is relatively slow.
Isn't there some sort of automated ping to the plugincheck website? Maybe we could look there for a sense of how many people have these versions to get a denominator?
Daniel - We can get a list of Flash numbers for a given OS via plugin-pings.
https://bugzilla.mozilla.org/show_bug.cgi?id=589912 comment #37 contains flash counts for a given date.

We could run a similar query for Windows and get total usage for a given flash version 10.1.x and 10.2.x based on Fx version + platform which should solve the population problem.

Then, run a MR job as per comment #2 or request chofmann to give us data for 2011-02-08 (Wednesday) as per comment #4 for crash counts.

Benjamin - thoughts?
That sounds great.
Turns out, we only have Flash pings for the Mac platform :(

Chofmann - is it possible to generate a crash report for flash 10.1 and 10.2 on the mac platform? 

If yes, then this will allow us to figure out the total flash population and # of crashes for mac which might help benjamin's analysis.
(In reply to comment #13)
> Turns out, we only have Flash pings for the Mac platform :(
> 
> Chofmann - is it possible to generate a crash report for flash 10.1 and 10.2 on
> the mac platform? 
> 
> If yes, then this will allow us to figure out the total flash population and #
> of crashes for mac which might help benjamin's analysis.


yes,  just restrict the signatures to something that matches ^FlashPlayer* and you will get just mac crashes.   I  guess that assumes that Mac uptake on new flash versions is the same there as it is on Windows.  That might not be a good assumption unless we understand more about how flash gets distributed on Mac.

does it come with OS updates there?

someone at adobe might help in providing some insight there.
Restricting the signatures is completely worthless: many of the crashes associated with Flash do not have flashplayer signatures. We should continue to use plugin-process+Flashplayerplugin limiting criteria (on 4beta builds).
benjamin - 
Based on your bug description, how about I run a MR job to get all crashes for mac os x based on debug id: 91A9F4C9E35FC25E987C8EFF94EB3A440 and "Flash Player" from the module list, get flash population count via plugin check and generate some numbers for MacOSX?

Note that the plugin check only happens if your existing flash version is older than the current one and not on every Firefox run. Not a 100% accurate metric, but it might just work here.

thoughts?
Why don't you get all Flash crashes and group them by debug ID?

I'm a little pessimistic right now about getting confidently-useful data since we don't have good population metrics. Maybe we should give up on the current data, and instead focus on whatever we need to do to get this information for the next set of releases.
Benjamin - I agree, no matter how much we marshal the data, it won't give accurate information on what you need.
I would suggest we open a new bug similar to https://bugzilla.mozilla.org/show_bug.cgi?id=589912 for tracking Flash pings on Windows.

lmk if this makes sense and i'll file it..
Yes, I think that makes sense. This is a report that we're going to want to run at every Flashplayer release, so we need to have good population numbers on a regular basis.
I added https://bugzilla.mozilla.org/show_bug.cgi?id=633554 as a blocker to this bug. Feel free to increase the priority.
Whiteboard: ETA: 2/10 for first iteration → waiting on 633554
I played with making a few reports that watch flash crash volume around the time of each release.  That makes some assumptions about the early flash beta test population being similar from release to release, as it its with firefox.

first set of numbers are at

http://people.mozilla.org/~chofmann/crash-stats/flash/flash-rollouts.txt

it shows crash volumes of 10.1.102.6 back in Nov. 2010, then some recent numbers for 10.1.102.6

these are follow by some numbers for 10.2.152.x which look like a variety of builds tested for the 10.2 release.

the numbers show that for early versions of 10.1 the number of reports coming in from firefox 3.6.x users was pretty high relative to 4.0beta testers.

for 10.2 crash rates for 4.0beta testers seem to be the same or higher as 3.6.x.

http://people.mozilla.org/~chofmann/crash-stats/flash/flash-10.2-sdiff-36-40.txt also shows that the higher volume from 4.0beta users doesn't seem to be concentrated on a few signatures, but it seems spreadout over a number of signatures.
(In reply to comment #14)
> (In reply to comment #13)
> > Turns out, we only have Flash pings for the Mac platform :(
> > 
> > Chofmann - is it possible to generate a crash report for flash 10.1 and 10.2 on
> > the mac platform? 
> > 
> > If yes, then this will allow us to figure out the total flash population and #
> > of crashes for mac which might help benjamin's analysis.
> 
> 
> yes,  just restrict the signatures to something that matches ^FlashPlayer* and
> you will get just mac crashes.   I  guess that assumes that Mac uptake on new
> flash versions is the same there as it is on Windows.  That might not be a good
> assumption unless we understand more about how flash gets distributed on Mac.
> 
> does it come with OS updates there?
> 
> someone at adobe might help in providing some insight there.

Apple has been pretty good about updates the the Flash Player but I would say the adoption rate is not as fast as Windows.  We are working on a auto update for Mac similar to Windows that should improve this.
Flash ping counts for Windows + Firefox + Flash. Pings give us good estimates on size of flash population. see bug - https://bugzilla.mozilla.org/show_bug.cgi?id=633554 for how we calculate flash pings.

windows & firefox 3.6.13 & flash 10.1.x
2011-02-20	161550 
2011-02-19	149055
2011-02-18	188709
2011-02-17	221982


windows & firefox 3.6.13 & flash 10.2.x
2011-02-20	77079 
2011-02-19	67204
2011-02-18	74541
2011-02-17	79769
(continuing wrt comment #23)
firefox 4.x & flash 10.1.x
2011-02-20	922
2011-02-19	964
2011-02-18	1079
2011-02-17	1086

firefox 4.x & flash 10.2.x
2011-02-20	837
2011-02-19	925
2011-02-18	839
2011-02-17	804
bsmedberg: do numbers in comment #23 and #24 make sense to proceed and also get numbers for flash crashes in comment #2?
Whiteboard: waiting on 633554 → waiting for reply on comment 25
Per IRC, the numbers do look reasonable.
Whiteboard: waiting for reply on comment 25 → writing MR job for OOPP Fx 4.0b crash wrt Flash 10.1 and 10.2 respectively
Whiteboard: writing MR job for OOPP Fx 4.0b crash wrt Flash 10.1 and 10.2 respectively → writing MR job for OOPP Fx 4.0b crashes wrt Flash 10.1 and 10.2 respectively
benjamin - do we have enough data to arrive to a conlusing and close this bug?
Whiteboard: writing MR job for OOPP Fx 4.0b crashes wrt Flash 10.1 and 10.2 respectively
I don't see the breakdown by signature. I'm not sure what the .csv files here are saying.
Based on comment #2, I ran a MR job to collect all 3.6.13 and flash 10.1.xx crashes for 4 days. The csv contains dates and the actual number of crashes for that date for given fx and flash version. All 10.1.xx crashes were summed up together as 10.1 crash, ditto for 10.2 
Were u looking for the full flash version (10.1.xx.xx) or something else?
No, I'm looking for a topcrash list grouped by crash signature for Flash 10.1 and Flash 10.2.

Hopefully where, instead of raw numbers of crashes, I get population percentages.

Then by comparing the two lists, I could see (as a made-up example) that the crash signature F_13245 is 5x more prevalent with Flash 10.2 than Flash 10.1
ignoring crashes < 0.01%
Attachment #514357 - Attachment is obsolete: true
Attachment #514517 - Attachment is obsolete: true
Attachment #515187 - Attachment is obsolete: true
The ADUs for Flash 10.x on Fx 4b11 are too small, 3101 and 2934 for 10.1 and 10.2 respectively. We didn't have this problem for Fx 3.6 as the numbers were significant enough. I am not sure how to tackle this issue, one option would be to wait for a week and get more data.
we should turn this into a regular report.  now we are looking for number of people testing the 10.3 beta.

we won't get adu numbers out of adobe so the only way to understand the migration patterns is tow watch them each day or each week release to release.
chofmann - in this case, is it okay if i close this bug and create a new one to count number of users on 10.3 beta split by platform + firefox version?
Comment 35 is a good report, if that was the question. Is there any reason we can't automate this to run every week with a breakdown by whatever Flash versions are found?
we can definitely automate this to run every week with a breakdown of flash version. As a note, we won't be able to accurately predict the Flash ADU's but come with an approximation of how many total users are using a given flash version.

Do you want to group the numbers for the entire week or by each day?
For the week ought to be fine.
Depends on: 630312
Whiteboard: depends on 630312
Laura - should this request be a part of socorro 1.7+ report?
Assignee: aphadke → laura
Component: Socorro → General
Product: Webtools → Socorro
Is this data request still relevant, i.e. does it still need to be done? If not, can you close the bug? Thanks.
Component: General → Data request
QA Contact: socorro → data-request
Yes, this is definitely still relevant. It is a request for an ongoing report that we can give to Adobe to compare the most recent release with the previous release of Flash. Obviously 10.1 and 10.2 specifically aren't still relevant, it's Flash 11 now.
If this should result in an ongoing report, we should probably put that in a new bug (as this already has enough comments with ad-hoc data gathering) and exactly spec out what the report should be - e.g. overall Flash crash/hang volume, crash vs. hang comparisons, rates per ADU, top signatures.
I'm running some custom reports on some pieces of that right now to monitor overall volume, crashes vs. hangs per Flash version, and top hang pairs. If we want something from that or something else to make it into Socorro, we need to exactly spec it and get separate bugs for everything the team needs to implement.

That said, bug 640241 is already on file and in the plans to get the Flash version reports that chofmann is running right now to be implemented in Socorro itself.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: