571696 - Add more breakdowns by report_type to /daily ADU report

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Austin King [:ozten]]

Copied from Bug#571526 Comment#2

I think events of the last few days show that we really nee to track all these
things:

just firefox crashes
firefox + plugin crashes
firefox + plugin crashes + uniq hang pairs

something like

https://bug571118.bugzilla.mozilla.org/attachment.cgi?id=450855

helps to draw the best picture of what is going on.

maybe a combo box is the bast way of thinking about how to add and remove lines
from the chart.

this is going to require extra work so maybe we spin this request off into
another bug and just keep this one focused on the quicker fix to change the
calculation to isolate just the firefox crashes.

Comment 1

[Austin King [:ozten]]

Current state of /daily

Any - any event that caused a crash (browser crash, unique hang id)
Crash - browser crash
Hang - browser hang, plugin hang - includes duplicate submission

Backend Data:
We also have OOPP crashes which aren't exposed in /daily

Comment 2

[Austin King [:ozten]]

Attachment: New web service aduByDayDetails to support graphing 1 or more crash report types

This adds a new web service which is similar to ADU API, but finer grained:
http://code.google.com/p/socorro/wiki/AduDetailsAPI

Also it pulls servicesList out of webservices config and into code.

Comment 3

[Austin King [:ozten]]

Attachment: Adds third state to /daily for graphing crashes by report type

Updates /daily to have a "Crashes per ADU by Report Type". This allows the user to get stats on any of
* crashes
* oopp
* hang report, browser part
* hang report, plugin part

This data can then be selectively plotted

Data table and CSV output is also updated.

Comment 4

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Comment on attachment 455017 [details] [diff] [review]
Adds third state to /daily for graphing crashes by report type

* formatURL() - Add hang_type to docs. Remove report type.
* prepareCrashGraphDataByReportType() - In docs, operating system should really be versions.
* Add a 1px border between Crashes by O/S and Crashes by Type on the ADU page
* In Crashes per ADU by Type, the Hang Plugin and Hang Browser data looks really close, and in the graphs they are indistinguishable. I don't think it's a problem with the code, just noting it.

Nice work.  Good consolidation and refactoring.  Thanks for shaping up the CSV methods.

Comment 5

[Laura Thomson :laura]

Comment on attachment 455017 [details] [diff] [review]
Adds third state to /daily for graphing crashes by report type

As we discussed in IRC, either refactor the throttling code out into a function, or put up some cautionary comments around it.  If you do the former I'll look at it again.

Comment 6

[Laura Thomson :laura]

Comment on attachment 455016 [details] [diff] [review]
New web service aduByDayDetails to support graphing 1 or more crash report types

The logic seems fine.  I'll leave it to lars to comment on the quality of your pythonic idiom.

Comment 7

[Austin King [:ozten]]

@lars - I need a code review on Attachment#455016 [details] [diff] Thanks.

Comment 8

[K Lars Lohn [:lars] [:klohn]]

Comment on attachment 455016 [details] [diff] [review]
New web service aduByDayDetails to support graphing 1 or more crash report types

looking through the code, I'm reminded about how much I love writing list comprehensions, but paradoxically hate reading them.

There is a "TODO" comment in there that I'm assuming that you intend to remain. 

While I approve the movement of the configuration parameter "servicesList" out of the configuration in this case, I've recently run across a use case where having it in the configuration is very fortuitous.  I won't go into detail here as it doesn't affect this code fragment migration.

Comment 9

[Austin King [:ozten]]

Filed Bug#580384. Once this is deployed, I'll check-in the frontend code.

Comment 10

[Austin King [:ozten]]

@chofmann or others, looking forward to feedback/testing.
http://crash-stats.stage.mozilla.com/daily

Click "ADU by Type" to graph by report type.

Comment 11

[Laura Thomson :laura]

This might not merge easily with the 1.7.5 changes.  Review and see if it's still necessary and how hard it is and then we'll make a decision.

Comment 12

[Laura Thomson :laura]

(In reply to comment #11)
> This might not merge easily with the 1.7.5 changes.  Review and see if it's
> still necessary and how hard it is and then we'll make a decision.

Still wanted.

Comment 13

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Integrated Austin's ADU work with all of the changes that have taken place since then.  Will be available for testing on stage shortly.

We will probably need to restart web services on stage to ensure that the new ADU API is available.

==

Sending        application/controllers/daily.php
Sending        application/models/daily.php
Adding         application/views/daily/daily_crash_data_by_report_type.php
Sending        application/views/daily/daily_search.php
Sending        application/views/daily/index.php
Sending        css/daily.css
Sending        js/socorro/daily.js
Transmitting file data .......
Committed revision 2949.

Comment 14

[Laura Thomson :laura]

(In reply to comment #13)

> 
> We will probably need to restart web services on stage to ensure that the new
> ADU API is available.
> 

Also need to pick up the config change.

Comment 15

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #14)
> (In reply to comment #13)
> 
> > 
> > We will probably need to restart web services on stage to ensure that the new
> > ADU API is available.
> > 
> 
> Also need to pick up the config change.

Ryan/Jabba, did we do this, yet?

Comment 16

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Yep.  If you click Crashes per ADU by Type on the left part of the ADU page, you'll get some results.  We still need to repopulate the stage database with ADU data, which we are waiting on currently.