444033 - Drop throttled/enabled percentage of Windows client to 10%

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Samuel Sidler (old account; do not CC)]

Spun out of bug 424960 (starting with comment 26).

We should drop the throttled percentage the crash reporter client uses to 10% from its current 25%.

This should really block 1.9.0.2 given the load on our servers.

Comment 1

[Michael Morgan [:morgamic]]

So Mac and Linux are completely unthrottled, correct?

Comment 2

[Michael Morgan [:morgamic]]

Ken - is there a healthy % that gets us the statistics we need from the Windows demographic?  What data would you need to determine that %?

Comment 3

[chris hofmann]

We should also try to compare active average daily users to unique users
reporting crashes or some other similar metric that might give us a handle on 
how many users are flipping the opt-in/opt-out checkbox.

One answer for the abnormal and unexpectedly high number of reports on firefox
3 is that a lot more people are opting in to send us crash data.   If that is
the case throttling back from 25% to 10% might also not have the desired effect
that we are seeking.  By giving users an easy to mark check box we might be
effectively DoDing socorro.

I know that some people may not like it but we might need to go to the system
we have used in the past to only offer crash reporting to 10% of users at
install time to do some effective throttling.   If a user missed out on the
random 10%, and they really want to get involved, they could also install the
extension to add breakpad.

Comment 4

[Michael Morgan [:morgamic]]

Makes sense to me.

Comment 5

[Reed Loden [:reed]]

(In reply to comment #3)
> If a user missed out on the
> random 10%, and they really want to get involved, they could also install the
> extension to add breakpad.

Breakpad isn't an extension, though. Not sure you can do that...

Comment 6

[chris hofmann]

jay and aravind were going to try to figure out and run some sql queries to dig out some cummulative since 3.0 release, and daily active crash reporters per day to help in some decision making for this bug.   Once we get a few samples of that data we can compare to AUS active daily users.

Comment 7

[(not currently active) Ted Mielczarek]

Attachment: reduce throttling percentage to 10% in firefox release config

Here's a simple patch to ratchet the number down to 10% from 25%. I'm with chofmann here though, I think we might have just made it too easy to opt-in. If wanted, I could put together a patch that would disable the crash reporter completely for X% of installs. I was thinking maybe just a file in the install dir, like "crashreporter.disabled", whose presence would cause us to not install the exception handler. If we do it in the installer, though, it won't help us throttle our existing userbase, or people coming from major update, for example. Open to ideas on that.

Comment 8

[chris hofmann]

We have also had a "disable yourself" feature in the past.

 Client contacts the server with a crash report.
 
 Server checks the client's build id and other info.
 
 If the server finds a match for some established criteria
 
   then send back a message to the client to don't send the crash report 
        and set crashreporter.disabled like ted proposes in comment 7

   else 
        regular processing


This is useful when we no longer want/need crash reports from older builds and need to do things like migrate to major changes in crash database schema or the entire system.

Do we have something like this or have this feature in the works?  If so we could also throttle back existing 3.0 release users with this feature.  If not I'll file another bug.

Comment 9

[(not currently active) Ted Mielczarek]

The client-side supports disabling by version, I implemented that in bug 412788. The server-side isn't implemented, that's bug 412790. However, the text for this disabling is probably not what you want for throttling, since it's geared for "end of life":
http://mxr.mozilla.org/mozilla-central/source/toolkit/locales/en-US/crashreporter/crashreporter.ini#55

We could certainly flip this for 3.0 once we do a few minor releases, however.

Comment 10

[Blake Cutler]

(In reply to comment #2)
> Ken - is there a healthy % that gets us the statistics we need from the Windows
> demographic?  What data would you need to determine that %?
> 

10% should be more than enough.  With a daily population size of 50k, you would attain a statistically significant result with a throttle percentage of 7.6%. 

Comment 11

[(not currently active) Ted Mielczarek]

Checked in, although I still think we need a stronger throttling solution at this point.

Comment 12

[chris hofmann]

> I was thinking maybe just a file in the install dir, 
> like "crashreporter.disabled", whose presence would cause us to not
> install the exception handler. 

should we branch off another bug, or continue on this one?

here is another idea that would allow us to 
1) not change RC bits after they are created
2) collect a 100% of crashes during release candidate testing period.
3) throttle back to only showing the crash dialog to 10% of users after the official is on the wire.

It would involve something along the lines of what ted suggested.  We could use a "born on date" on the client installation to figure out whether the client should send all reports or become part of the 90% of installations that don't sent reports for final releases.  To do something like this we would need files or prefs to test against where we had run checks to figure out if the client is in the 10% random pool, and if the crashreporter_is.disabled or the crashreporter_is.enabled.   Maybe change the name of the crashreporter.disabled file to crashreporter.participation and then insert "enabled" or "disabled" as contents of the file.   Here is a rough outline of what the logic might look like.


if crashreporter.participation file doesn't exist set one up

   if build date < 20 days ago
      echo "enabled" > crashreporter.participation

   if build date > 20 days ago
   
      if in the random 10% pool
           echo "enabled" > crashreporter.participation
      else 
           echo "disabled" > crashreporter.participation


 if crash.participation file contains "enabled"
   show the crash dialog and let the user send in the crash data
 else 
   don't install the exception handler and/or show the crash dialog to the user.


That would give us a 20 day window after the builds are created as 100% reporting, then we would start to set up a random 90% of clients as "disabled" after that.

It would also make it easy for advanced users to "turn on crash reporting" if they really wanted to, or to shut it down if they knew how and where to go find this file and edit the contents.

Comment 13

[Samuel Sidler (old account; do not CC)]

(In reply to comment #12)
> should we branch off another bug, or continue on this one?

Move it to another bug, please.

Comment 14

[chris hofmann]

ok, it is bug 444961