974726 - implement correlations in postgresql

Status: RESOLVED FIXED

(Socorro :: Database, task)

Description

[Robert Helmer [:rhelmer]]

Now that we have processed_crashes JSON in postgres, we should be able to implement correlation reports which automatically track current product and versions, replacing the old python scripts at http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools/

Comment 1

[Robert Helmer [:rhelmer]]

This seems to work, but I am not sure if I'm doing the right thing here, especially in the addons and modules reports: https://gist.github.com/rhelmer/9107425

Every crash has 0 or more addons, and 0 or more modules, in JSON arrays. The CPU core count is simpler since there's always one CPU type and a single number-of-cores per crash.

The intent is that we'd build matviews with the output of these, which could be used to reproduce the current reports:
https://crash-analysis.mozilla.com/crash_analysis/20140219/20140219_Firefox_26.0-interesting-addons.txt.gz
https://crash-analysis.mozilla.com/crash_analysis/20140219/20140219_Firefox_26.0-interesting-modules.txt.gz
https://crash-analysis.mozilla.com/crash_analysis/20140219/20140219_Firefox_26.0-core-counts.txt.gz

There is also "with versions" variants of the addons/modules reports, I think it'd be easy to just  ignore the versions (or not) when querying the matviews.

Comment 2

[Selena Deckelmann :selenamarie :selena]

Here's one small optimization: https://gist.github.com/selenamarie/9122115

Once we have this going, we can probably add a few indexes on json data. Let's give it a whirl for a week and then come back for optimizations separately.

Great work, :rhelmer! This is very exciting.

Comment 3

[Robert Helmer [:rhelmer]]

PR: https://github.com/mozilla/socorro/pull/1908

Comment 4

[Robert Helmer [:rhelmer]]

Note that this is just covering the stored procedures and matviews - let's tackle the crontabber and mware/django bits in separate bugs. I think it's valuable to land this first so we can backfill on stage and play with the data.

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/960e64fd44d12ea622696c9cdc139aaff0d27723
fixes bug 974726 - implement correlations using postgres

https://github.com/mozilla/socorro/commit/dfffb54b46249285623e0612706418ce11cd5940
Merge pull request #1908 from rhelmer/bug974726-correlations-using-postgres

fixes bug 974726 - implement correlations using postgres

Comment 6

[Selena Deckelmann :selenamarie :selena]

*throws confetti*

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/a58bc77134086f3fce85ad1628a1ad0b53552efc
bug 974726 - drop old update_correlatinos function

https://github.com/mozilla/socorro/commit/717d75d2b22d10cf6f6192925abda5ebf5e818b4
Merge pull request #1956 from rhelmer/bug974726-drop-old-update_correlations

Bug974726 drop old update correlations