Last Comment Bug 640286 - Extract stats about Gfx features from crash reports, monitor success/failure/crashes
: Extract stats about Gfx features from crash reports, monitor success/failure/...
Status: NEW
:
Product: Core
Classification: Components
Component: Graphics (show other bugs)
: unspecified
: x86_64 Linux
: -- normal with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
Depends on: 639620 640341
Blocks:
  Show dependency treegraph
 
Reported: 2011-03-09 12:35 PST by Benoit Jacob [:bjacob] (mostly away)
Modified: 2012-02-05 12:44 PST (History)
5 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
churn_crash_reports.sh (3.15 KB, text/plain)
2011-03-09 12:35 PST, Benoit Jacob [:bjacob] (mostly away)
no flags Details
churn_crash_reports.sh (3.48 KB, text/plain)
2011-03-09 13:27 PST, Benoit Jacob [:bjacob] (mostly away)
no flags Details
Stats from March 10 to March 25 (584.30 KB, text/plain)
2011-03-28 05:26 PDT, Benoit Jacob [:bjacob] (mostly away)
no flags Details
c++ gfx-stats program (5.99 KB, text/plain)
2012-02-04 21:45 PST, Benoit Jacob [:bjacob] (mostly away)
no flags Details
c++ gfx-stats program, with help showing how to run in a script (7.38 KB, text/plain)
2012-02-05 12:44 PST, Benoit Jacob [:bjacob] (mostly away)
no flags Details

Description Benoit Jacob [:bjacob] (mostly away) 2011-03-09 12:35:10 PST
Created attachment 518148 [details]
churn_crash_reports.sh

Since bug 627464 we have information about tried/successful/denied Gfx features (D2D, Layers, WebGL...).

The attach script uses that to automatically extract a bunch of information, specifically:
 * stats of how many % of users actually get each feature
 * stats of users who crash during initialization of each feature

You first need to download crashdata.csv files from:
  http://people.mozilla.org/crash_analysis/

The you run the script. If you want to get info about all Gfx features on all platforms, do:

  ./churn_crash_reports.sh all all *.csv

more generally the calling syntax is

  ./churn_crash_reports.sh FILTER FEATURE *.csv

Where 'all' can be used for FILTER and for FEATURE.

FILTER is a regular expression used to grep the crash reports so for example Windows.NT.6 will filter for Windows Seven/Vista.

FEATURE can be one of: 'D3D9 Layers' 'D3D10 Layers' 'GL Layers' D2D DWrite WebGL


You will notice that D2D and DWrite show very high success rates: that's because they are only attempted if D3D10 Layers are successful.



Sample run on crash data from March 3 to March 8:

bjacob@cahouette:~$ ./churn_crash_reports.sh all all *.csv
===== Statistics for D3D9 Layers =====
On 20110303, 205 tried D3D9 Layers, 11 (5%) succeeded
On 20110304, 504 tried D3D9 Layers, 50 (9%) succeeded
On 20110305, 653 tried D3D9 Layers, 60 (9%) succeeded
On 20110306, 605 tried D3D9 Layers, 75 (12%) succeeded
On 20110307, 724 tried D3D9 Layers, 92 (12%) succeeded
On 20110308, 773 tried D3D9 Layers, 82 (10%) succeeded, 1 (0%) crashed trying

Crash reports during D3D9 Layers initialization:
http://crash-stats.mozilla.com/report/index/10f14385-f526-43c2-9c02-47f702110308  Windows NT 6  Intel

===== Statistics for D3D10 Layers =====
On 20110303, 266 tried D3D10 Layers, 49 (18%) succeeded
On 20110304, 725 tried D3D10 Layers, 184 (25%) succeeded
On 20110305, 938 tried D3D10 Layers, 267 (28%) succeeded
On 20110306, 919 tried D3D10 Layers, 259 (28%) succeeded
On 20110307, 1033 tried D3D10 Layers, 306 (29%) succeeded
On 20110308, 1075 tried D3D10 Layers, 281 (26%) succeeded

===== Statistics for GL Layers =====
On 20110303, 24 tried GL Layers, 14 (58%) succeeded
On 20110304, 57 tried GL Layers, 30 (52%) succeeded
On 20110305, 25 tried GL Layers, 14 (56%) succeeded
On 20110306, 26 tried GL Layers, 23 (88%) succeeded
On 20110307, 62 tried GL Layers, 62 (100%) succeeded
On 20110308, 34 tried GL Layers, 34 (100%) succeeded

===== Statistics for D2D =====
On 20110303, 52 tried D2D, 49 (94%) succeeded
On 20110304, 216 tried D2D, 191 (88%) succeeded
On 20110305, 283 tried D2D, 267 (94%) succeeded, 2 (0%) crashed trying
On 20110306, 309 tried D2D, 292 (94%) succeeded
On 20110307, 342 tried D2D, 322 (94%) succeeded, 4 (1%) crashed trying
On 20110308, 316 tried D2D, 299 (94%) succeeded

Crash reports during D2D initialization:
http://crash-stats.mozilla.com/report/index/7b965549-6328-45b2-8eab-02c312110305  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/ac6d4851-67cb-4b84-a1da-94a142110305  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/ae995730-76a0-476b-88a8-216c82110307  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/2b255fd9-a491-444b-859d-d24ed2110307  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/e5fed9be-4edc-49cc-9592-5fe012110307  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/3927cf90-800e-455e-869c-c62b12110307  Windows NT 6  ATI

===== Statistics for DWrite =====
On 20110303, 55 tried DWrite, 55 (100%) succeeded
On 20110304, 199 tried DWrite, 199 (100%) succeeded
On 20110305, 274 tried DWrite, 274 (100%) succeeded
On 20110306, 296 tried DWrite, 295 (99%) succeeded, 1 (0%) crashed trying
On 20110307, 325 tried DWrite, 324 (99%) succeeded
On 20110308, 311 tried DWrite, 308 (99%) succeeded, 3 (0%) crashed trying

Crash reports during DWrite initialization:
http://crash-stats.mozilla.com/report/index/924d5f67-9baf-4037-9c64-28c952110306  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/65003809-aaf8-48ce-8ea1-0babd2110308  Windows NT 6  Intel
http://crash-stats.mozilla.com/report/index/02cc61a7-51f3-4a75-990e-a53ce2110308  Windows NT 6  Intel
http://crash-stats.mozilla.com/report/index/9b1a396c-48c9-4f0f-9483-e61f72110308  Windows NT 6  Intel

===== Statistics for WebGL =====
On 20110303, 24 tried WebGL, 16 (66%) succeeded
On 20110304, 65 tried WebGL, 49 (75%) succeeded, 4 (6%) crashed trying
On 20110305, 47 tried WebGL, 24 (51%) succeeded, 1 (2%) crashed trying
On 20110306, 83 tried WebGL, 16 (19%) succeeded, 1 (1%) crashed trying
On 20110307, 81 tried WebGL, 57 (70%) succeeded, 3 (3%) crashed trying
On 20110308, 57 tried WebGL, 37 (64%) succeeded, 4 (7%) crashed trying

Crash reports during WebGL initialization:
http://crash-stats.mozilla.com/report/index/a0ecb3cc-36cf-4b8d-b31d-fa7982110304  Linux
http://crash-stats.mozilla.com/report/index/87da2aff-034e-4d90-b5e4-b745e2110304  Linux
http://crash-stats.mozilla.com/report/index/6f93dc43-4fee-429e-857e-aa1b02110304  Linux
http://crash-stats.mozilla.com/report/index/e58c41a4-f04d-4770-8d57-b29472110304  Linux
http://crash-stats.mozilla.com/report/index/6c64e6c8-72e9-4f30-8bbc-4ba872110305  Linux
http://crash-stats.mozilla.com/report/index/5092cd05-9e0a-4767-aed6-2f9432110306  ATI
http://crash-stats.mozilla.com/report/index/27030891-fccf-4068-b22a-5c0dd2110307  Linux
http://crash-stats.mozilla.com/report/index/4ce65cbc-209f-4629-9498-25ef12110307  Linux
http://crash-stats.mozilla.com/report/index/604340d6-6f6b-4fa0-8179-d26a62110307  Linux
http://crash-stats.mozilla.com/report/index/2e22a25d-f09c-4b42-97e4-3ff062110308  Linux
http://crash-stats.mozilla.com/report/index/74594bb6-1800-4f82-8953-bf74e2110308  Linux
http://crash-stats.mozilla.com/report/index/a3f90655-7d95-432b-b61b-910932110308  Linux
http://crash-stats.mozilla.com/report/index/12c07cda-71ef-40df-a200-9500d2110308  Linux
Comment 1 Benoit Jacob [:bjacob] (mostly away) 2011-03-09 13:27:07 PST
Created attachment 518160 [details]
churn_crash_reports.sh

This updated version detects Optimus. It finds that the above-mentioned D3D9 crash was on Optimus.
Comment 2 Bas Schouten (:bas.schouten) 2011-03-09 15:56:50 PST
(In reply to comment #0)
> You will notice that D2D and DWrite show very high success rates: that's
> because they are only attempted if D3D10 Layers are successful.

This is incorrect, we only use D3D10 layers when D2D initialization is successful, however, I know what is going on:

1) For D3D10, you're getting a good measurement of how many of the sample set gets D2D/D3D10. Since we start a measurement at the start of the function, and we always call this function, even if we don't have a D3D10/D2D device created by gfxWindowsPlatform, we simply bail early if that device doesn't exist. See http://mxr.mozilla.org/mozilla-central/source/gfx/layers/d3d10/LayerManagerD3D10.cpp#141 and http://mxr.mozilla.org/mozilla-central/source/widget/src/windows/nsWindow.cpp#3447

2) For D2D, you're sample set for the measurement -only- devices that are not blocked. This means they have drivers less than half a year/9 months old. This sample set will most likely consist mainly of DX10 compliant cards, and in the majority of cases be successful. See http://mxr.mozilla.org/mozilla-central/source/gfx/thebes/gfxWindowsPlatform.cpp#316 and http://mxr.mozilla.org/mozilla-central/source/gfx/thebes/gfxWindowsPlatform.cpp#369

Also note that VerifyD2DDevice might succeed multiple times per session, and usually will succeed at least twice.
Comment 3 Benoit Jacob [:bjacob] (mostly away) 2011-03-11 05:42:50 PST
Thanks Bas for the explanation.

Here's now the results for the first full day of RC1:

On 20110310, 23029 tried D3D9 Layers, 2512 (10%) succeeded, 19 (0%) crashed trying
On 20110310, 28797 tried D3D10 Layers, 5064 (17%) succeeded, 1 (0%) crashed trying
On 20110310, 806 tried GL Layers, 801 (99%) succeeded, 2 (0%) crashed trying
  (the 2 GL layers crashes there are Windows/Intel, so it must be
   someone who played with non-default prefs.)
On 20110310, 5816 tried D2D, 5224 (89%) succeeded, 8 (0%) crashed trying
On 20110310, 5263 tried DWrite, 5234 (99%) succeeded, 25 (0%) crashed trying
On 20110310, 492 tried WebGL, 301 (61%) succeeded, 17 (3%) crashed trying

Will file bugs about these new initialization crashes.
Comment 4 Benoit Jacob [:bjacob] (mostly away) 2011-03-28 05:26:13 PDT
Created attachment 522339 [details]
Stats from March 10 to March 25
Comment 5 Benoit Jacob [:bjacob] (mostly away) 2012-02-04 21:45:33 PST
Created attachment 594510 [details]
c++ gfx-stats program

This tool can be used to generate the data at
http://people.mozilla.org/~bjacob/gfx_features_stats/
Comment 6 Benoit Jacob [:bjacob] (mostly away) 2012-02-05 12:44:44 PST
Created attachment 594582 [details]
c++ gfx-stats program, with help showing how to run in a script

This new version adds a --stdin-with-date option that allows to run much faster in a script feeding it with gzip -cd through a pipe, and the help is improved to in particular give such a script you can use to process all .csv.gz crashdata files in a directory, without uncompressing to disk. Here it is; it is given in --help anyway:

#!/bin/sh

for f in `ls *-pub-crashdata.csv.gz`
do
  echo "$f"
  gzip -dc $f | ./gfx-stats --stdin-with-date $f
done

Note You need to log in before you can comment on or make changes to this bug.