Open Bug 640286 Opened 9 years ago Updated 3 years ago

Extract stats about Gfx features from crash reports, monitor success/failure/crashes

Categories

(Core :: Graphics, defect)

x86_64
Linux
defect
Not set

Tracking

()

People

(Reporter: bjacob, Unassigned)

References

Details

Attachments

(3 files, 2 obsolete files)

Attached file churn_crash_reports.sh (obsolete) —
Since bug 627464 we have information about tried/successful/denied Gfx features (D2D, Layers, WebGL...).

The attach script uses that to automatically extract a bunch of information, specifically:
 * stats of how many % of users actually get each feature
 * stats of users who crash during initialization of each feature

You first need to download crashdata.csv files from:
  http://people.mozilla.org/crash_analysis/

The you run the script. If you want to get info about all Gfx features on all platforms, do:

  ./churn_crash_reports.sh all all *.csv

more generally the calling syntax is

  ./churn_crash_reports.sh FILTER FEATURE *.csv

Where 'all' can be used for FILTER and for FEATURE.

FILTER is a regular expression used to grep the crash reports so for example Windows.NT.6 will filter for Windows Seven/Vista.

FEATURE can be one of: 'D3D9 Layers' 'D3D10 Layers' 'GL Layers' D2D DWrite WebGL


You will notice that D2D and DWrite show very high success rates: that's because they are only attempted if D3D10 Layers are successful.



Sample run on crash data from March 3 to March 8:

bjacob@cahouette:~$ ./churn_crash_reports.sh all all *.csv
===== Statistics for D3D9 Layers =====
On 20110303, 205 tried D3D9 Layers, 11 (5%) succeeded
On 20110304, 504 tried D3D9 Layers, 50 (9%) succeeded
On 20110305, 653 tried D3D9 Layers, 60 (9%) succeeded
On 20110306, 605 tried D3D9 Layers, 75 (12%) succeeded
On 20110307, 724 tried D3D9 Layers, 92 (12%) succeeded
On 20110308, 773 tried D3D9 Layers, 82 (10%) succeeded, 1 (0%) crashed trying

Crash reports during D3D9 Layers initialization:
http://crash-stats.mozilla.com/report/index/10f14385-f526-43c2-9c02-47f702110308  Windows NT 6  Intel

===== Statistics for D3D10 Layers =====
On 20110303, 266 tried D3D10 Layers, 49 (18%) succeeded
On 20110304, 725 tried D3D10 Layers, 184 (25%) succeeded
On 20110305, 938 tried D3D10 Layers, 267 (28%) succeeded
On 20110306, 919 tried D3D10 Layers, 259 (28%) succeeded
On 20110307, 1033 tried D3D10 Layers, 306 (29%) succeeded
On 20110308, 1075 tried D3D10 Layers, 281 (26%) succeeded

===== Statistics for GL Layers =====
On 20110303, 24 tried GL Layers, 14 (58%) succeeded
On 20110304, 57 tried GL Layers, 30 (52%) succeeded
On 20110305, 25 tried GL Layers, 14 (56%) succeeded
On 20110306, 26 tried GL Layers, 23 (88%) succeeded
On 20110307, 62 tried GL Layers, 62 (100%) succeeded
On 20110308, 34 tried GL Layers, 34 (100%) succeeded

===== Statistics for D2D =====
On 20110303, 52 tried D2D, 49 (94%) succeeded
On 20110304, 216 tried D2D, 191 (88%) succeeded
On 20110305, 283 tried D2D, 267 (94%) succeeded, 2 (0%) crashed trying
On 20110306, 309 tried D2D, 292 (94%) succeeded
On 20110307, 342 tried D2D, 322 (94%) succeeded, 4 (1%) crashed trying
On 20110308, 316 tried D2D, 299 (94%) succeeded

Crash reports during D2D initialization:
http://crash-stats.mozilla.com/report/index/7b965549-6328-45b2-8eab-02c312110305  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/ac6d4851-67cb-4b84-a1da-94a142110305  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/ae995730-76a0-476b-88a8-216c82110307  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/2b255fd9-a491-444b-859d-d24ed2110307  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/e5fed9be-4edc-49cc-9592-5fe012110307  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/3927cf90-800e-455e-869c-c62b12110307  Windows NT 6  ATI

===== Statistics for DWrite =====
On 20110303, 55 tried DWrite, 55 (100%) succeeded
On 20110304, 199 tried DWrite, 199 (100%) succeeded
On 20110305, 274 tried DWrite, 274 (100%) succeeded
On 20110306, 296 tried DWrite, 295 (99%) succeeded, 1 (0%) crashed trying
On 20110307, 325 tried DWrite, 324 (99%) succeeded
On 20110308, 311 tried DWrite, 308 (99%) succeeded, 3 (0%) crashed trying

Crash reports during DWrite initialization:
http://crash-stats.mozilla.com/report/index/924d5f67-9baf-4037-9c64-28c952110306  Windows NT 6  ATI
http://crash-stats.mozilla.com/report/index/65003809-aaf8-48ce-8ea1-0babd2110308  Windows NT 6  Intel
http://crash-stats.mozilla.com/report/index/02cc61a7-51f3-4a75-990e-a53ce2110308  Windows NT 6  Intel
http://crash-stats.mozilla.com/report/index/9b1a396c-48c9-4f0f-9483-e61f72110308  Windows NT 6  Intel

===== Statistics for WebGL =====
On 20110303, 24 tried WebGL, 16 (66%) succeeded
On 20110304, 65 tried WebGL, 49 (75%) succeeded, 4 (6%) crashed trying
On 20110305, 47 tried WebGL, 24 (51%) succeeded, 1 (2%) crashed trying
On 20110306, 83 tried WebGL, 16 (19%) succeeded, 1 (1%) crashed trying
On 20110307, 81 tried WebGL, 57 (70%) succeeded, 3 (3%) crashed trying
On 20110308, 57 tried WebGL, 37 (64%) succeeded, 4 (7%) crashed trying

Crash reports during WebGL initialization:
http://crash-stats.mozilla.com/report/index/a0ecb3cc-36cf-4b8d-b31d-fa7982110304  Linux
http://crash-stats.mozilla.com/report/index/87da2aff-034e-4d90-b5e4-b745e2110304  Linux
http://crash-stats.mozilla.com/report/index/6f93dc43-4fee-429e-857e-aa1b02110304  Linux
http://crash-stats.mozilla.com/report/index/e58c41a4-f04d-4770-8d57-b29472110304  Linux
http://crash-stats.mozilla.com/report/index/6c64e6c8-72e9-4f30-8bbc-4ba872110305  Linux
http://crash-stats.mozilla.com/report/index/5092cd05-9e0a-4767-aed6-2f9432110306  ATI
http://crash-stats.mozilla.com/report/index/27030891-fccf-4068-b22a-5c0dd2110307  Linux
http://crash-stats.mozilla.com/report/index/4ce65cbc-209f-4629-9498-25ef12110307  Linux
http://crash-stats.mozilla.com/report/index/604340d6-6f6b-4fa0-8179-d26a62110307  Linux
http://crash-stats.mozilla.com/report/index/2e22a25d-f09c-4b42-97e4-3ff062110308  Linux
http://crash-stats.mozilla.com/report/index/74594bb6-1800-4f82-8953-bf74e2110308  Linux
http://crash-stats.mozilla.com/report/index/a3f90655-7d95-432b-b61b-910932110308  Linux
http://crash-stats.mozilla.com/report/index/12c07cda-71ef-40df-a200-9500d2110308  Linux
Attached file churn_crash_reports.sh
This updated version detects Optimus. It finds that the above-mentioned D3D9 crash was on Optimus.
Attachment #518148 - Attachment is obsolete: true
Depends on: 640341
Depends on: 639620
(In reply to comment #0)
> You will notice that D2D and DWrite show very high success rates: that's
> because they are only attempted if D3D10 Layers are successful.

This is incorrect, we only use D3D10 layers when D2D initialization is successful, however, I know what is going on:

1) For D3D10, you're getting a good measurement of how many of the sample set gets D2D/D3D10. Since we start a measurement at the start of the function, and we always call this function, even if we don't have a D3D10/D2D device created by gfxWindowsPlatform, we simply bail early if that device doesn't exist. See http://mxr.mozilla.org/mozilla-central/source/gfx/layers/d3d10/LayerManagerD3D10.cpp#141 and http://mxr.mozilla.org/mozilla-central/source/widget/src/windows/nsWindow.cpp#3447

2) For D2D, you're sample set for the measurement -only- devices that are not blocked. This means they have drivers less than half a year/9 months old. This sample set will most likely consist mainly of DX10 compliant cards, and in the majority of cases be successful. See http://mxr.mozilla.org/mozilla-central/source/gfx/thebes/gfxWindowsPlatform.cpp#316 and http://mxr.mozilla.org/mozilla-central/source/gfx/thebes/gfxWindowsPlatform.cpp#369

Also note that VerifyD2DDevice might succeed multiple times per session, and usually will succeed at least twice.
Thanks Bas for the explanation.

Here's now the results for the first full day of RC1:

On 20110310, 23029 tried D3D9 Layers, 2512 (10%) succeeded, 19 (0%) crashed trying
On 20110310, 28797 tried D3D10 Layers, 5064 (17%) succeeded, 1 (0%) crashed trying
On 20110310, 806 tried GL Layers, 801 (99%) succeeded, 2 (0%) crashed trying
  (the 2 GL layers crashes there are Windows/Intel, so it must be
   someone who played with non-default prefs.)
On 20110310, 5816 tried D2D, 5224 (89%) succeeded, 8 (0%) crashed trying
On 20110310, 5263 tried DWrite, 5234 (99%) succeeded, 25 (0%) crashed trying
On 20110310, 492 tried WebGL, 301 (61%) succeeded, 17 (3%) crashed trying

Will file bugs about these new initialization crashes.
Attached file c++ gfx-stats program (obsolete) —
This tool can be used to generate the data at
http://people.mozilla.org/~bjacob/gfx_features_stats/
Attachment #594510 - Attachment mime type: text/x-c++src → text/plain
This new version adds a --stdin-with-date option that allows to run much faster in a script feeding it with gzip -cd through a pipe, and the help is improved to in particular give such a script you can use to process all .csv.gz crashdata files in a directory, without uncompressing to disk. Here it is; it is given in --help anyway:

#!/bin/sh

for f in `ls *-pub-crashdata.csv.gz`
do
  echo "$f"
  gzip -dc $f | ./gfx-stats --stdin-with-date $f
done
Attachment #594510 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.