629055 - Highlight explosive crashes in crash lists

Status: RESOLVED INVALID

(Socorro :: Webapp, task)

Description

[Laura Thomson :laura]

As per
https://wiki.mozilla.org/Socorro:PRD_2.x#New_.2F_explosive_.2F_critical_crash_tracking

There's a mock at
https://wiki.mozilla.org/File:Explosive_signatures.png

We'll need to get this data into the database, first.  (Separate bug)

Comment 1

[[:jberkus] Josh Berkus]

The data already exists in the database.  I just need to give Ryan some query-writing help. Ryan, let me know when you want it.

Comment 2

[Laura Thomson :laura]

Josh - this is a UI bug, you want bug 629062.

Comment 3

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Right, we'll need to have an API call in the Pythonic middleware layer that serves this data up.  We are no longer adding new db calls on the PHP side.  Is the API call requirement implicit in bug 629062?

Comment 4

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

@Chofmann - What defines an explosive crash?

Per the spec, I have:
Explosive crashes are ones that have experienced a sharp upward change in frequency. Exact formula needs to be developed, combining increase in volume and rank. 

Perhaps, I can recommend these as a place to start.  All of the following should be highlighted in this list:
1. A crash signature that is within the top 100 topcrashers for a particular version, and this is the only version in which this crash signature has appeared.
2. A crash signature that has a % change in difference from the previous top crashers period of > 1%.
3. A crash signature that has risen over 50 spots in top crasher rank since the previous top crasher period.

Comment 5

[[:jberkus] Josh Berkus]

> Perhaps, I can recommend these as a place to start.  All of the following
> should be highlighted in this list:
> 1. A crash signature that is within the top 100 topcrashers for a particular
> version, and this is the only version in which this crash signature has
> appeared.

Seems doable.

> 2. A crash signature that has a % change in difference from the previous top
> crashers period of > 1%.

This would get a lot of false positives.  If there were 2 crashes yesterday, and 3 today, that would be an increase of 50%.  I think we need an increase of more than 100% *and* more than 100 crashes, or something similar.

> 3. A crash signature that has risen over 50 spots in top crasher rank since the
> previous top crasher period.

The above should only apply if the new rank is in the top 100 (or maybe top 50).  It might be just as effective to say, instead:

Signature is in the top 50 today, and was not in the top 100 yesterday.

Comment 6

[Laura Thomson :laura]

(In reply to comment #3)
> Right, we'll need to have an API call in the Pythonic middleware layer that
> serves this data up.  We are no longer adding new db calls on the PHP side.  Is
> the API call requirement implicit in bug 629062?


I think that's ok.

Comment 7

[chris hofmann]

there is a set of 14 bugs in this bugzilla query that show examples of explosive crashes that have happened over the last several months.

https://bugzilla.mozilla.org/buglist.cgi?field0-0-5=content&type0-0-4=substring&value0-0-5=%22explosive%22&type0-0-5=matches&keywords=crash&keywords_type=allwords&field0-0-0=product&type0-0-1=substring&field0-0-1=component&field0-0-4=status_whiteboard&value0-0-2=explosive&type0-0-3=substring&query_format=advanced&value0-0-3=explosive&field0-0-3=short_desc&value0-0-4=explosive&field0-0-2=alias&value0-0-1=explosive&type0-0-0=substring&value0-0-0=explosive&type0-0-2=substring

there are probably several more that we could dig out connected to other external events like web site updates,  plugin releases, and our own firefox product regressions.

the goal of the explosive crash effort its to detect these kinds of problem earlier, provide a warning system to those "on-watch" for finding new crashes, and then finally to streamline the analysis so we can connect the new crash to those three kinds of events.

some research back though the old explosive crash bug list can help us to determine reasonable hourly crash volume rates and ranking changes that go beyond the norms where pagers and e-mail messages need to be triggered.

Comment 8

[Laura Thomson :laura]

Blocking bug is slipping to 1.7.8, so this goes too.

Comment 9

[Laura Thomson :laura]

Other priorities are higher for now.

Comment 10

[Robert Kaiser]

No sense to target this to before when we even detect explosive crashes ;-)

Comment 11

[Peter Bengtsson [:peterbe]]

Is this a dup of #629062 now?

Comment 12

[Robert Kaiser]

(In reply to Peter Bengtsson [:peterbe] from comment #11)
> Is this a dup of #629062 now?

Not unless that one adds some highlighting of explosive crashes into TCBS.

Comment 13

[Peter Bengtsson [:peterbe]]

Explosive crashes is deprecated.