845970 - Reverse correlation report: crashes which highly correlate with AMD graphics cards

Status: RESOLVED WORKSFORME

(Socorro :: Data request, task)

Description

[Benjamin Smedberg]

Another report related to AMD graphics and bug 844156/bug 772330. Currently we check correlations when we see a topcrash. But I suspect that we may actually have these crashes in other builds, but they are affecting code locations which run less often, or cause crashes where the signature is more random.

What I'd like to do is have a report for each beta and release which spits out the signature of any crash signature which correlates highly with AMD drivers, whether or not it is a topcrash.

Limiting factor:

* windows-only
* For now, correlate only on AdapterVendorID: this is in postgres in the app_notes field, or directly in the raw annotations in hbase

There is no point in running this report for nightly or Aurora: the signatures are likely to change on every build, and the populations there aren't likely large enough to give us good statistics for single days.

This does need to be a report specific to a single build. I suggest we run it:

1 week after each beta is released, over the data from the release date (1 week)
1, 2, and 4 weeks after a release, over the data from the past week

Comment 1

[Benjamin Smedberg]

Oh sorry, important datapoint: for the AMD crashes in question, the AdapterVendorID is usually "0x1002".

Comment 2

[Selena Deckelmann :selenamarie :selena]

Working on this today. I have a set of queries from the last correlations I worked on that can mostly be pressed into service. 

Should have some data by about noon PT today.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Attachment: Version 16.0 correlations - signature to AdapterVendorID: 0x1002

Comment 4

[Selena Deckelmann :selenamarie :selena]

A little analysis on the data I gathered

signature F1398665248_____________________________ really jumped out.


selena=# select signature, product_version,total_crashes, crash_by_signature_and_vendorid, "P(sig, vendid) / (P(sig) * P(vendid))" from crashes_by_amd where crash_by_signature_and_vendorid > 100 order by "P(sig, vendid) / (P(sig) * P(vendid))" desc; 
-[ RECORD 1 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | F1650855128_____________________________
product_version                       | 20.0
total_crashes                         | 12803
crash_by_signature_and_vendorid       | 108
P(sig, vendid) / (P(sig) * P(vendid)) | 4.667517317
-[ RECORD 2 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | @0x0 | __RtlUserThreadStart | _RtlUserThreadStart
product_version                       | 20.0
total_crashes                         | 12803
crash_by_signature_and_vendorid       | 106
P(sig, vendid) / (P(sig) * P(vendid)) | 1.492998807
-[ RECORD 3 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | @0x0 | __RtlUserThreadStart | _RtlUserThreadStart
product_version                       | 19.0
total_crashes                         | 33303
crash_by_signature_and_vendorid       | 169
P(sig, vendid) / (P(sig) * P(vendid)) | 1.232001035
-[ RECORD 4 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | F1398665248_____________________________
product_version                       | 18.0.2
total_crashes                         | 4175
crash_by_signature_and_vendorid       | 464
P(sig, vendid) / (P(sig) * P(vendid)) | 1.162656008
-[ RECORD 5 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | F1531689212_______________________________________________________________________________________
product_version                       | 19.0
total_crashes                         | 33303
crash_by_signature_and_vendorid       | 133
P(sig, vendid) / (P(sig) * P(vendid)) | 1.143797300
-[ RECORD 6 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | F1398665248_____________________________
product_version                       | 19.0.1
total_crashes                         | 756
crash_by_signature_and_vendorid       | 120
P(sig, vendid) / (P(sig) * P(vendid)) | 1.122744487
-[ RECORD 7 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | F_11935219________________________________________________
product_version                       | 19.0
total_crashes                         | 33303
crash_by_signature_and_vendorid       | 165
P(sig, vendid) / (P(sig) * P(vendid)) | 1.075617654
-[ RECORD 8 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | F1398665248_____________________________
product_version                       | 19.0
total_crashes                         | 33303
crash_by_signature_and_vendorid       | 3071
P(sig, vendid) / (P(sig) * P(vendid)) | 1.067251971
-[ RECORD 9 ]-------------------------+---------------------------------------------------------------------------------------------------
signature                             | F2005414273____________________________________________________________________________________
product_version                       | 19.0
total_crashes                         | 33303
crash_by_signature_and_vendorid       | 192
P(sig, vendid) / (P(sig) * P(vendid)) | 1.026301914
-[ RECORD 10 ]------------------------+---------------------------------------------------------------------------------------------------
signature                             | F1398665248_____________________________
product_version                       | 20.0
total_crashes                         | 12803
crash_by_signature_and_vendorid       | 921
P(sig, vendid) / (P(sig) * P(vendid)) | 0.977953676
-[ RECORD 11 ]------------------------+---------------------------------------------------------------------------------------------------
signature                             | F_855855466_____________________
product_version                       | 19.0
total_crashes                         | 33303
crash_by_signature_and_vendorid       | 186
P(sig, vendid) / (P(sig) * P(vendid)) | 0.903953422

Comment 5

[Selena Deckelmann :selenamarie :selena]

Attachment: SQL dump of a table containing the correlations

Comment 6

[Selena Deckelmann :selenamarie :selena]

Attachment: Python script to generate SQL for finding the correlations

I'm attaching the quick and dirty script I used to limit the time spent on the secondary server. These queries are quite lengthy when run together, but are ok if limited to one version at a time. Looking forward to having a reporting server where long running queries don't set off alarms!!

Comment 7

[Robert Kaiser]

Selena, the F* signatures are Flash crashes, which are mostly nothing we ourselves can do anything about, so while they are an interesting data point, I think Benjamin also wants to know about others. Also, why are the P() values larger than 1?

Comment 8

[Selena Deckelmann :selenamarie :selena]

(In reply to Robert Kaiser (:[e-mail]) from comment #7)
> why are the P() values larger than 1?

See https://bugzilla.mozilla.org/show_bug.cgi?id=797068#c19 for a detailed explanation of the proability ratio.

Comment 9

[Benjamin Smedberg]

* There's absolutely no need to run this on multiple versions: we'll know externally what release or beta we're trying to test.
* This is primarily for browser crashes, not plugin crashes, so I'm going to remove the limiters.

Have you committed these scripts to any repos? I'll play with them and stick them in socorro-toolbox unless you have a better place for them.

Comment 10

[Benjamin Smedberg]

Back to me, using https://github.com/bsmedberg/socorro-toolbox/compare/correlationscripts to generate better data now.

Comment 11

[Benjamin Smedberg]

I can run these as-needed.