587758 - make an admin panel for plugging in default values for throttle rates on crash per 100 ADU chart

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[chris hofmann]

We tried to do something simple for the crash per 100 user chart at
http://crash-stats.mozilla.com/daily?p=Firefox&v[]=

the first stage was to pull together all the data needed to build the graph and provide a simple graphing tool that could be used to compare releases and that has  worked ok for people with intimate knowledge of throttling levels on client and server for each of the recent releases.   

the problem is that people outside of that group are a also trying to use the charting tool, and getting incorrect results out of the system.

we need to provide a few more enhancements so a wider group can start to use the charting tool to get more accurate results.

In the original implementation we thought about trying to make this work off the processor config file that controls throttling on the server side, and mabe the history of any changes to that file, but that adds lots of complexity, and still doesn't account for older releases that have client side throttling.

one way we could do this is to build an admin interface that would pre-fill the correct throttling level for a give release instead of defaulting to 100% for all releases.

e.g. the admin panel would allow these vaules to be plugged in and used as defaults.

release  default_throttle_value

 3.5.11  3.5%
 3.5.12  3.5%

 3.6.8   10%


Once someone with admin access plugs these numbers in, then they would be used as the default throttling level when generating a chart for that release.

The default would be used when generating the graph for any of the given releases, and could be overridden if someone wanted to experiment with new throttle assumptions, or an older period where the throttle value might have been different.

Comment 1

[chris hofmann]

laura, this is probably going to need some attention as we get closer to to shipping firefox 4 and more people are going to be looking at the chart to figure out to figure out if firefox 4 is as stable or better than  3.6.x and 3.5.x

we should figure out which socorro release we can make some progress on this bug.

Comment 2

[Laura Thomson :laura]

All right.  Should we hold 1.7.5 for this?

Comment 3

[chris hofmann]

it the next release is in a couple of weeks we can proably hold off and do this then.

Comment 4

[Laura Thomson :laura]

Next release after 1.7.5 is likely to be mid-end November.  Still ok?

Comment 5

[chris hofmann]

yeah, there will probably more people wanting to compare 4.0 betas against 3.5 and 3.6 starting around when b7 and b8 get a lot of users.  that most likely will be around mid nov.

Comment 6

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

@chofmann Is there a list of the throttle values for existing versions?  Is there a preferred default value that we should enter into the throttle field?

Comment 7

[chris hofmann]

there is a config file that contains the current throttle values used by the server.  one option would be to just work off that as the starting point. arvind can tell you where the config file lives so you could look at the format of the file.

Basically we have throttling set at 10% for all the major shipping releases.

3.5.x and earlier releases also have 10% of users opt'ed in to sending in crash reports, so for those the throttle rate is 10% of 10% or 1%.   We suspect maybe another 1% or 2% override their default opt-out setting so for 3.5.x and 3.0.x we should set the default throttle rate to be about 2%.

If we could just plug those values in quickly that would be a good first step.  Then we could do fancier admin panel stuff to adjust these basic values if needed.

Comment 8

[Aravind Gottipati [:aravind]]

This is the current setting.

$config['throttle_rates'] = array(
    'Firefox' => array(
        '3.0.15' => 15,
        '3.0.16' => 15,
        '3.0.17' => 15,
        '3.0.18' => 15,
        '3.0.19' => 15,
        '3.0.20' => 15,
        '3.0.21' => 15,
        '3.5.5' => 15,
        '3.5.6' => 15,
        '3.5.7' => 15,
        '3.5.8' => 15,
        '3.5.9' => 15
    )

Comment 9

[chris hofmann]

so with 3.5.x its 10% client side throttling plus another 10% that might over ride the opt-out, then server side is 15% of those reports.

3.6.x should be in the list too.  

also, where do we keep that config file in version control?   is it web accessible?

Comment 10

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

The file is in webapp-php/application/config/daily.php, which is a config file that only a sysadmin can touch.  When we built the original ADU charts, we thought that would be the best way to control default throttle settings.

The admin panel will provide us with the ability to backfill throttle rates, so even if we don't get them all now, we can update them later.  I'll include these rates as default settings for the release.

Comment 11

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Attachment: Patch 1 for 587758

This patch provides support for adding a default throttle rate to each version.  It requires the addition of a new field to the product_visibility table - 'throttle'.  

In the branches admin page when adding/updating versions, a throttle field is now available for editing the throttle percentage.  Meanwhile, in the ADU table, when a version is selected from the dropdown box in the ADU menu, the throttle value specific to that version will automatically be inserted in the throttle field.

Comment 12

[Robert Helmer [:rhelmer]]

Comment on attachment 484976 [details] [diff] [review]
Patch 1 for 587758

LGTM. One nit (indentation):

>Index: webapp-php/application/controllers/daily.php
(...)
>         if ( isset($parameters['hang_type'])) {
> 	    $hang_type = $parameters['hang_type'];
> 	} else {
>-            $hang_type = 'any';
>+        $hang_type = 'any';
> 	}

Comment 13

[Laura Thomson :laura]

Also note re comment #8: those webapp values are *wrong* in prod, so misleading.  It's 10% for all the released versions in the collector, and 100% for betas, specials, anything with an email address, TB, SM, and Camino.

Comment 14

[Laura Thomson :laura]

Comment on attachment 484976 [details] [diff] [review]
Patch 1 for 587758

What about history?   Consider the typical example at the moment where we get various betas and RCs of 3.6.foo.  When 3.6.foo is released, we throttle it down to 10%.  This patch doesn't cover that use case, so graphs for 3.6.foo will have an irregularity where the throttle rate changes.

I think we probably need to store throttle history in a database table to produce realistic graphs.  

It would also be good to make sure whatever value we're using in the webapp is synced with the collector config.

In summary: I have r-ed this patch, but if chofmann is ok with these constraints, you can move ahead with it since it's better than what we have now.  Can we annotate the graphs with the throttle value being used?  I think that would make it more transparent to the user.

Comment 15

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Thanks for the review Rob.

Thanks Laura for bringing that point up.

In IRC, Laura, Chofmann and I discussed whether or not this would be a good enough solution to carry us for a while.  Ultimately, we decided it would be best to get this solution out the door.  

But we will need to communicate that throttle rates could have changed in any point during that time and may have affected the graphs.  And we need to document in Google Code how to determine what the effective throttling rate is for admins.

Feel free to pitch in if there is anything else I need to include here.

Also, creating bug 606294 to ensure this gets integrated and tested properly in 1.8.

Comment 16

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

@Choffman, please edit the wiki entry for throttle rates if it can be stated more clearly.
http://code.google.com/p/socorro/wiki/SocorroUIAdmin#Throttle

Comment 17

[chris hofmann]

probably way to wordy, but I update to cover all the main points.

Comment 18

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Thanks Choffman.

Committing these changes.  Stage update request is in Bug 606387.  Updated 1.7.5 release on Google Code wiki.

==

Sending        socorro/database/schema.py
Sending        webapp-php/application/config/daily.php-dist
Sending        webapp-php/application/controllers/admin.php
Sending        webapp-php/application/controllers/daily.php
Sending        webapp-php/application/controllers/products.php
Sending        webapp-php/application/models/branch.php
Sending        webapp-php/application/views/admin/branch_data_sources.php
Sending        webapp-php/application/views/daily/daily_search.php
Sending        webapp-php/application/views/daily/index.php
Sending        webapp-php/css/daily.css
Sending        webapp-php/js/socorro/admin.js
Sending        webapp-php/js/socorro/daily.js
Transmitting file data ............
Committed revision 2638.

Comment 19

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Fixing one minor issue found when loading the page without a selected product.
- Committed revision 2642.

==

This is available for testing on stage.

To test, you may change the throttle value associated with a version via the branches admin at:
http://crash-stats.stage.mozilla.com/admin/branch_data_sources

Then you may test the results by switching to that version on the ADU page.  When you select a version, it will update the throttle value associated with that version.  The user may then override the throttle value manually.
http://crash-stats.stage.mozilla.com/daily?p=Firefox

Comment 20

[chris hofmann]

marcia, christian, tomcat...  we need to add a step when setting up products to enter a default throttling value for each release so people stop being confused about crashes per 100 users.   can you check this out and see if it works for you?