572419 - Investigate changes in crash correlated plugins since the spike

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Laura Thomson :laura]

sayrer has a couple of possible theories:
* A popular plugin issued an update that is crashing us more in 3.6.4
* Flash zero day infected more users with Trojans

Can you please dig into the correlations and see if you find data that would indicate either of these scenarios?

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I don't know of any data sources I have access to that would allow doing this.  

Also, how would this happen without significant changes in crash signature composition?  And who has the power to push updates to all our Mac and Windows users that take effect in < 1 hour?

Comment 2

[Laura Thomson :laura]

I think we can do it from Hbase so Xavier can help out.

What I'm interested in:
plugins including versions where known by frequency before the spike / after the spike
extensions before the spike/after the spike

Rob, if there's other data you want to see please add it to this bug.

Comment 3

[Laura Thomson :laura]

dbaron: agree that it's unlikely, but we should cross all the ts and dot the is.

Comment 4

[Xavier Stevens [:xstevens]]

I think I have something ready for this, but we need to get HBase in production stabilized so I can run a job on that data.

My job is setup to calculate counts for each day in a specified time window (I plan to use June 1st - June 14th and June 6th - June 9th) based on all combinations of os, sig, module, module_version.  It then calculates min, max, mean, std. dev., variance and skewness.

We will be able to sort the results by one of these columns and get different looks into the data.  Standard deviation and skewness will probably be the more useful than the other stats at seeing what spiked during this period.

Comment 5

[Laura Thomson :laura]

xstevens, ozten: status?

Comment 6

[Xavier Stevens [:xstevens]]

I'm running my job this morning.  Ran into some memory issues now that I have all the data.  Should have something before noon though.

Comment 7

[Xavier Stevens [:xstevens]]

Attachment: Crash Trends for 6/6/2010 to 6/9/2010 sorted by growth

I produced crashtrends.growth which the format is:

os:signature:module:version<tab>min<tab>max<tab>avg<tab>std. dev.<tab>growth

growth is percentage calculated like so:

if min == 0:
    growth = (max - min) * 100
else:
    growth = ((max - min) / min) * 100

The attached file is currently sorted by growth.  Let me know if you see anything we think we should investigate further.

Comment 8

[Xavier Stevens [:xstevens]]

Looking at this further.  Growth is a bad metric the way I have calculated it. 
Some of the top entries spiked on 1 day and then gone again the subsequent two
days.

Comment 9

[Xavier Stevens [:xstevens]]

Attachment: Crash Counts 6/6/2010 to 6/9/2010

Format of the file:

os_name:signature:module:version<tab><6/6 count><tab><6/7 count>...<tab><6/10 count>

Comment 10

[Xavier Stevens [:xstevens]]

Looking over the results to me it just looks like crashes are up across the board.  Asked other metrics/socorro to take a look as well, but I'm not sure what else I can do at this point.

Comment 11

[Laura Thomson :laura]

I think we're done here - results don't have any revelations.  Thanks for helping rule this out Xavier.