Closed Bug 632937 Opened 14 years ago Closed 13 years ago

Need comparison of Flash 10.1 and Flash 10.2 crash rates and topcrashes (trunk and branch)

Categories

(Socorro :: Data request, task)

Product:
Socorro
Component:
Data request
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: benjamin, Assigned: laura)

References

Details

(Whiteboard: depends on 630312)

Attachments

(2 files, 3 obsolete files)

firefox3.6, crashes grouped by flash 10.1 and flash 10.2
51.64 KB, text/csv
Details
firefox4.0b11, crashes grouped by flash 10.1 and flash 10.2. This file is only for reference as the ADUs for Flash 10.x on Fx4b11 cannot be correctly determined.
217.23 KB, text/csv
Details
I need some custom topcrash reports made available separated by Flash 10.1 and Flash 10.2. I think this is going to require some map-reduce jobs, because the Flash version is not stored anywhere else in the database. The purpose of these reports is to take equivalent versions of Firefox (3.6.13 and 4.0b11, say) and see what crashes have been fixed in Flash 10.2, and which new ones have been introduced. Because adoption of Flash may ramp up slowly, I'm not exactly sure how to get comparable numbers, I'd love statistical suggestions here. On Windows, the version of Flash is reported in the module list under NPSWF32.dll. On Mac, the version of Flash is also in the module list, but it's a bit more complicated. Flash 10.2 is listed as "FlashPlayer-10.6", doesn't have a version entry, but does have a debug ID of 91A9F4C9E35FC25E987C8EFF94EB3A440. For older versions of Flash, the module is "Flash Player". I think that if we group the results by the debug ID, that should be sufficient. This is a fairly high priority, I'd like preliminary results by Monday if possible.
Need to see how well aphadke can do this analysis since xstevens will be on PTO the rest of the week.
Assignee: nobody → aphadke
paraphrasing IRC conversations between benjamin and anurag: Build a list of all signatures for Flash crashes on Firefox 3.6.13/Flash 10.1 and Firefox 3.6.13/Flash 10.2 Restrict to Windows for first iteration. Flash version can be found by looking at: NPSWF32.dll on frame 0 of raw_dump (or crash_signature inside HBase) Look for corresponding flash version in the module list: Module|NPSWF32.dll|10.2.152.26|NPSWF32.pdb|AE9732643EEF42A7B7DBF7BC85F2EDB21|0x5fcb0000|0x60296fff|0 10.2.152.26 is the flash version, we are interested in 10.2.xxx Compare via percentage as 10.2 population might be small. Repeat for: Firefox 4.0b11/Flash10.1 and Firefox 4.0b11/Flash 10.2
Whiteboard: ETA: 2/10 for first iteration
flash version info is in the .csv files @ field 22 ./format.sh 20110207* 1 signature 2 url 3 uuid_url 4 client_crash_date 5 date_processed 6 last_crash 7 product 8 version 9 build 10 branch 11 os_name 12 os_version 13 cpu_info 14 address 15 bug_list 16 user_comments 17 uptime_seconds 18 email 19 adu_count 20 topmost_filenames 21 addons_checked 22 flash_version 23 hangid 24 reason 25 process_type 26 app_notes simple command to show counts of all firefox and flash version pairs for all the NPSWF32.dll signatures on 2011 02 07 looks like awk -F\t '$1 ~ /NPSWF32.dll/ {print $8,$22}' 20110207* | sort | uniq -c | sort -nr | more 1275 3.6.13 10.0.32.18 1174 3.6.13 10.0.42.34 1065 3.6.13 10.0.22.87 876 3.6.13 10.0.12.36 451 3.5.16 10.0.32.18 285 3.6.13 10.2.161.23 262 3.5.16 10.0.22.87 260 3.5.16 10.0.42.34 247 3.5.16 10.0.12.36 188 4.0b10 10.0.22.87 155 4.0b10 10.0.32.18 153 4.0b10 10.0.12.36 142 3.0.19 10.0.22.87 140 3.6 10.0.42.34 135 4.0b10 10.0.42.34 121 3.6.13 10.0.2.54 119 4.0b10 10.2.152.14 117 3.6.13 10.2.152.14 114 3.0.19 10.0.12.36 113 3.6.13 10.1.102.64 106 4.0b10 10.2.161.23 104 3.6.13 9.0.124.0 104 3.0.19 10.0.32.18 Data for each day is in the pub-crashdata files at places like http://people.mozilla.org/crash_analysis/20110208/
quick look shows relatively the same numbers of flash 10.1 and 10.2 crashes for all NPSWF32.dll signatures on feb7 for 3.6.13 but 3x more for flash 10.2 on 4.0b10 flash10.1 flash10.2 3.6.13 337 440 4.0b10 77 299 4.0b9 5 12
those numbers seem to hold up looking at a larger sample for feb 1-7 join flash10.1 flash10.2 3.6.13 2257 2829 4.0b10 487 2110 4.0b9 59 100 4.0b9pre 1 1
the regex might need a bit of refinement but if I just add $1 ~ /^F/ to pick up all the flash crashes like 11094 F957328252________________________ 1048 F_1441563101_______________________ 987 F_437410032______________________________________________________________ 947 FFPlugin.dll@0x19d0c4 666 F1146177332__________________________ 491 FlashPlayer-10.6@0x2ace83 408 FFPlugin.dll@0x173ee4 392 F1559406643________________________________________ 368 F_1115197261____________________________________________________________ I get different results where total flash crashes on 10.2 are only 23-25% compared to 10.1 across both firefox 3.13 and 4.b10 join flash10.1 flash10.2 3.6.13 171929 40182 23% 4.0b10 33412 8361 25% 3.6.3 5452 3053 3.6 4546 2965 3.6.8 3238 848 4.0b9 1883 537 4.0b7 1637 474 3.6.6 1537 432 4.0b8 1461 383 there is a bit of noise in the matches for /^F/ but not much.
Are the higher numbers for overall NPSWF32.dll crashes on 10.2 and firefo 4 just a possible indication that we need some symbols for specific releases of flash and a few people in the beta population just happen to be on those versions of flash in higher relative numbers than the general public? Looks like the concentration is on flash 10.2.161.23 and 10.2.152.14 counts ff flash version 285 3.6.13 10.2.161.23 106 4.0b10 10.2.161.23 117 3.6.13 10.2.152.14 119 4.0b10 10.2.152.14 113 3.6.13 10.1.102.64 11 4.0b10 10.1.102.64 66 3.6.13 10.1.51.66 44 3.6.13 10.1.51.45 18 4.0b10 10.1.51.45 40 3.6.13 10.1.51.95 29 3.6.13 10.2.152.21 68 4.0b10 10.2.152.21 21 3.6 10.1.51.95 10 4.0b10 10.1.51.95 18 3.6.13 10.1.52.14 26 4.0b12pre 10.2.152.14 16 3.6.13 10.1.53.7 12 3.6.13 10.1.53.64 10 3.6.13 10.1.85.3 10 3.6.12 10.2.161.23 11 4.0b12pre 10.2.161.23 10 4.0b9 10.2.161.23
Considering that Flash 10.2 final was not released until Tuesday afternoon, I really don't think numbers before 8-Feb are going to be that interesting. I'm interested in all the plugin crashes with the Flash plugin-container, not just the /^F/ crashes. The hard part of this is figuring out what part of the population has Flash 10.1 and Flash 10.2: you really can't compare just the number of crashes, or we won't know whether the total volume has gone up significantly. Flash adoption is relatively slow.
Isn't there some sort of automated ping to the plugincheck website? Maybe we could look there for a sense of how many people have these versions to get a denominator?
Daniel - We can get a list of Flash numbers for a given OS via plugin-pings. https://bugzilla.mozilla.org/show_bug.cgi?id=589912 comment #37 contains flash counts for a given date. We could run a similar query for Windows and get total usage for a given flash version 10.1.x and 10.2.x based on Fx version + platform which should solve the population problem. Then, run a MR job as per comment #2 or request chofmann to give us data for 2011-02-08 (Wednesday) as per comment #4 for crash counts. Benjamin - thoughts?
That sounds great.
Turns out, we only have Flash pings for the Mac platform :( Chofmann - is it possible to generate a crash report for flash 10.1 and 10.2 on the mac platform? If yes, then this will allow us to figure out the total flash population and # of crashes for mac which might help benjamin's analysis.
(In reply to comment #13) > Turns out, we only have Flash pings for the Mac platform :( > > Chofmann - is it possible to generate a crash report for flash 10.1 and 10.2 on > the mac platform? > > If yes, then this will allow us to figure out the total flash population and # > of crashes for mac which might help benjamin's analysis. yes, just restrict the signatures to something that matches ^FlashPlayer* and you will get just mac crashes. I guess that assumes that Mac uptake on new flash versions is the same there as it is on Windows. That might not be a good assumption unless we understand more about how flash gets distributed on Mac. does it come with OS updates there? someone at adobe might help in providing some insight there.
Restricting the signatures is completely worthless: many of the crashes associated with Flash do not have flashplayer signatures. We should continue to use plugin-process+Flashplayerplugin limiting criteria (on 4beta builds).
benjamin - Based on your bug description, how about I run a MR job to get all crashes for mac os x based on debug id: 91A9F4C9E35FC25E987C8EFF94EB3A440 and "Flash Player" from the module list, get flash population count via plugin check and generate some numbers for MacOSX? Note that the plugin check only happens if your existing flash version is older than the current one and not on every Firefox run. Not a 100% accurate metric, but it might just work here. thoughts?
Why don't you get all Flash crashes and group them by debug ID? I'm a little pessimistic right now about getting confidently-useful data since we don't have good population metrics. Maybe we should give up on the current data, and instead focus on whatever we need to do to get this information for the next set of releases.
Benjamin - I agree, no matter how much we marshal the data, it won't give accurate information on what you need. I would suggest we open a new bug similar to https://bugzilla.mozilla.org/show_bug.cgi?id=589912 for tracking Flash pings on Windows. lmk if this makes sense and i'll file it..
Yes, I think that makes sense. This is a report that we're going to want to run at every Flashplayer release, so we need to have good population numbers on a regular basis.
I added https://bugzilla.mozilla.org/show_bug.cgi?id=633554 as a blocker to this bug. Feel free to increase the priority.
Whiteboard: ETA: 2/10 for first iteration → waiting on 633554
I played with making a few reports that watch flash crash volume around the time of each release. That makes some assumptions about the early flash beta test population being similar from release to release, as it its with firefox. first set of numbers are at http://people.mozilla.org/~chofmann/crash-stats/flash/flash-rollouts.txt it shows crash volumes of 10.1.102.6 back in Nov. 2010, then some recent numbers for 10.1.102.6 these are follow by some numbers for 10.2.152.x which look like a variety of builds tested for the 10.2 release. the numbers show that for early versions of 10.1 the number of reports coming in from firefox 3.6.x users was pretty high relative to 4.0beta testers. for 10.2 crash rates for 4.0beta testers seem to be the same or higher as 3.6.x. http://people.mozilla.org/~chofmann/crash-stats/flash/flash-10.2-sdiff-36-40.txt also shows that the higher volume from 4.0beta users doesn't seem to be concentrated on a few signatures, but it seems spreadout over a number of signatures.
(In reply to comment #14) > (In reply to comment #13) > > Turns out, we only have Flash pings for the Mac platform :( > > > > Chofmann - is it possible to generate a crash report for flash 10.1 and 10.2 on > > the mac platform? > > > > If yes, then this will allow us to figure out the total flash population and # > > of crashes for mac which might help benjamin's analysis. > > > yes, just restrict the signatures to something that matches ^FlashPlayer* and > you will get just mac crashes. I guess that assumes that Mac uptake on new > flash versions is the same there as it is on Windows. That might not be a good > assumption unless we understand more about how flash gets distributed on Mac. > > does it come with OS updates there? > > someone at adobe might help in providing some insight there. Apple has been pretty good about updates the the Flash Player but I would say the adoption rate is not as fast as Windows. We are working on a auto update for Mac similar to Windows that should improve this.
Flash ping counts for Windows + Firefox + Flash. Pings give us good estimates on size of flash population. see bug - https://bugzilla.mozilla.org/show_bug.cgi?id=633554 for how we calculate flash pings. windows & firefox 3.6.13 & flash 10.1.x 2011-02-20 161550 2011-02-19 149055 2011-02-18 188709 2011-02-17 221982 windows & firefox 3.6.13 & flash 10.2.x 2011-02-20 77079 2011-02-19 67204 2011-02-18 74541 2011-02-17 79769
(continuing wrt comment #23) firefox 4.x & flash 10.1.x 2011-02-20 922 2011-02-19 964 2011-02-18 1079 2011-02-17 1086 firefox 4.x & flash 10.2.x 2011-02-20 837 2011-02-19 925 2011-02-18 839 2011-02-17 804
bsmedberg: do numbers in comment #23 and #24 make sense to proceed and also get numbers for flash crashes in comment #2?
Whiteboard: waiting on 633554 → waiting for reply on comment 25
Per IRC, the numbers do look reasonable.
Whiteboard: waiting for reply on comment 25 → writing MR job for OOPP Fx 4.0b crash wrt Flash 10.1 and 10.2 respectively
Whiteboard: writing MR job for OOPP Fx 4.0b crash wrt Flash 10.1 and 10.2 respectively → writing MR job for OOPP Fx 4.0b crashes wrt Flash 10.1 and 10.2 respectively
benjamin - do we have enough data to arrive to a conlusing and close this bug?
Whiteboard: writing MR job for OOPP Fx 4.0b crashes wrt Flash 10.1 and 10.2 respectively
I don't see the breakdown by signature. I'm not sure what the .csv files here are saying.
Based on comment #2, I ran a MR job to collect all 3.6.13 and flash 10.1.xx crashes for 4 days. The csv contains dates and the actual number of crashes for that date for given fx and flash version. All 10.1.xx crashes were summed up together as 10.1 crash, ditto for 10.2 Were u looking for the full flash version (10.1.xx.xx) or something else?
No, I'm looking for a topcrash list grouped by crash signature for Flash 10.1 and Flash 10.2. Hopefully where, instead of raw numbers of crashes, I get population percentages. Then by comparing the two lists, I could see (as a made-up example) that the crash signature F_13245 is 5x more prevalent with Flash 10.2 than Flash 10.1
ignoring crashes < 0.01%
Attachment #514357 - Attachment is obsolete: true
Attachment #514517 - Attachment is obsolete: true
Attachment #515187 - Attachment is obsolete: true
The ADUs for Flash 10.x on Fx 4b11 are too small, 3101 and 2934 for 10.1 and 10.2 respectively. We didn't have this problem for Fx 3.6 as the numbers were significant enough. I am not sure how to tackle this issue, one option would be to wait for a week and get more data.
we should turn this into a regular report. now we are looking for number of people testing the 10.3 beta. we won't get adu numbers out of adobe so the only way to understand the migration patterns is tow watch them each day or each week release to release.
chofmann - in this case, is it okay if i close this bug and create a new one to count number of users on 10.3 beta split by platform + firefox version?
Comment 35 is a good report, if that was the question. Is there any reason we can't automate this to run every week with a breakdown by whatever Flash versions are found?
we can definitely automate this to run every week with a breakdown of flash version. As a note, we won't be able to accurately predict the Flash ADU's but come with an approximation of how many total users are using a given flash version. Do you want to group the numbers for the entire week or by each day?
For the week ought to be fine.
Depends on: 630312
Whiteboard: depends on 630312
Laura - should this request be a part of socorro 1.7+ report?
Assignee: aphadke → laura
Component: Socorro → General
Product: Webtools → Socorro
Is this data request still relevant, i.e. does it still need to be done? If not, can you close the bug? Thanks.
Component: General → Data request
QA Contact: socorro → data-request
Yes, this is definitely still relevant. It is a request for an ongoing report that we can give to Adobe to compare the most recent release with the previous release of Flash. Obviously 10.1 and 10.2 specifically aren't still relevant, it's Flash 11 now.
If this should result in an ongoing report, we should probably put that in a new bug (as this already has enough comments with ad-hoc data gathering) and exactly spec out what the report should be - e.g. overall Flash crash/hang volume, crash vs. hang comparisons, rates per ADU, top signatures. I'm running some custom reports on some pieces of that right now to monitor overall volume, crashes vs. hangs per Flash version, and top hang pairs. If we want something from that or something else to make it into Socorro, we need to exactly spec it and get separate bugs for everything the team needs to implement. That said, bug 640241 is already on file and in the plans to get the Flash version reports that chofmann is running right now to be implemented in Socorro itself.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: