909572 - Re-enable explosive report in web UI

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Shuhao]

Currently the explosive report is disabled when it lands.

This needs to be manually enabled by reverse a git commit or uncommenting some code.

Comment 1

[Lonnen :lonnen]

disabled here means unlinked, not waffled off.

Comment 2

[Shuhao]

I've been randomly sampling the thing thus far. It seems fairly stable. 

Should we enable it?

Comment 3

[Peter Bengtsson [:peterbe]]

What do you think Kairo & Laura?

Comment 4

[Robert Kaiser]

Well, I surely think we need to have the report public to evaluate it, and we definitely want to do that and find out how it fares in comparison to the one in my custom reports.

Comment 5

[Shuhao]

Sure in that case I can submit a PR. It should be really trivial.

Comment 6

[Shuhao]

PR: https://github.com/mozilla/socorro/pull/1626

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/7e6d50048342c2f09580ad8141f5fbe30fd57fbf
[Bug 909572] Re-enabled explosive crash stats

https://github.com/mozilla/socorro/commit/69b6689b7780f4d0adc60b937987efc6ad612e59
Merge pull request #1626 from shuhaowu/explosive-ui-909572

[Bug 909572] Re-enabled explosive crash stats

Comment 8

[Peter Bengtsson [:peterbe]]

The Eagle has landed. 


And by Eagle I mean the link.
And by landed I mean, landed on master.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Is there a way I can test this in time for shipping, potentially, on Wednesday, the 6th?

On https://crash-stats.allizom.org/explosive/products/Firefox, I see "Explosive crashes over the last 5 days," which I'm assuming means a cron or report/job will need to be run, first, before we see anything.

Comment 10

[Shuhao]

This cron job has already been running for the past 2 months. I've been periodically checking on its progress to make sure it is relatively stable.

This patch simply enables the link in the web UI and allows for easy access to the page (where as previously you will have to go and manually seek it out).

There are some other features that we should/can add to this, but this should serve as a good first thing to roll out to get some feedback.

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Shuhao Wu [:Pwnna] from comment #10)
> This cron job has already been running for the past 2 months. I've been
> periodically checking on its progress to make sure it is relatively stable.
> 
> This patch simply enables the link in the web UI and allows for easy access
> to the page (where as previously you will have to go and manually seek it
> out).
> 
> There are some other features that we should/can add to this, but this
> should serve as a good first thing to roll out to get some feedback.

Thx, but I might be missing something, then?  https://crash-stats.allizom.org/explosive/products/Firefox shows no "no explosive crashes found" over the last 5 days.

Selena, are we potentially missing a cronjob/something else being run?

Comment 12

[Shuhao]

That is normal if the system does not find any explosive crashes within the last couple of days.

The previous one detected was on Oct 28th (this one actually affected me) and it is reported at 931794.

The detector is set on settings that will only report on "explosive crashes" if the confidence exceeds 99.99% (however that number is likely skewed high..)

Comment 13

[Shuhao]

Also, refer to https://bugzilla.mozilla.org/show_bug.cgi?id=909575 for allow querying for longer ranges of time.

Comment 14

[Matt Brandt [:mbrandt]]

(In reply to Shuhao Wu [:Pwnna] from comment #13)
> Also, refer to https://bugzilla.mozilla.org/show_bug.cgi?id=909575 for allow
> querying for longer ranges of time.

Does this mean that QA will be unable to verify this bug until the ability to query for larger date ranges is enabled (bug 909575)?

Comment 15

[Shuhao]

You can still verify this bug. The larger date range is a potential feature.

If it shows no explosive crashes are found, it doesn't mean that the cron job is not running (although that could be a possibility, albeit small one given my monitoring of the /explosive url for the last two months), it just means that it hasn't found anything peculiar (which would be a good thing).

There could be other crashes that may show up as explosive anytime. If you check http://crash-stats.mozilla.com/explosive, you may be able to see some in the coming days..

Comment 16

[Matt Brandt [:mbrandt]]

(In reply to Shuhao Wu [:Pwnna] from comment #15)
> There could be other crashes that may show up as explosive anytime. If you
> check http://crash-stats.mozilla.com/explosive, you may be able to see some
> in the coming days..

Pwnna, sorry. Let me apologize for poorly framing my question. I am attempting to verify this feature on stage before it lands on production (it is marked as [qa+]). Can we infuse stage with data to enable verification prior to going to production?

Comment 17

[Shuhao]

Testing this is somewhat complicated as there are multiple components to this feature. (None of that is covered by this bug, though).

There's a cron job in the back that collects reports from reports_clean and count them up. It does some math and if it finds something explosive, it inserts a record into postgres. The test case for that is here[1] and I suspect that you can generate data similar to that on stage.

There's a middleware that pulls the data from postgres (table suspicious_crash_signatures). This can be pretty easily simulated and there are test cases here[2].

The final component is the web ui, which just gets data from the middleware. 

So if you want to test the first and second component on stage, you need /enough/ fake data as the parameters for the cronjob is tuned to fit the volume we see in production. Fake data is generally not sufficient to test if a system that essentially rely on real data (it is doing statistics on them so idealized situations obviously will work) is working or not by definition. To really test this, we need to be able to generate realistic looking data, which itself is a difficult task.

Keep in mind that this particular bug simply enables a link and none of what I just said.

[1]: https://github.com/mozilla/socorro/blob/master/socorro/unittest/cron/jobs/test_suspicious_crashes.py 
[2]: https://github.com/mozilla/socorro/blob/master/socorro/unittest/external/postgresql/test_suspicious.py