528437 - Unthrottle Thunderbird crash reports

Status: RESOLVED WORKSFORME

(Socorro :: General, task)

Comment 1

[Ludovic Hirlimann [:Usul]]

We are in the release process of 3.0 and just published Thunderbird 3.0rc1 builds for testing puposes.

This morning I used the crash-extension to test that crash reports where correctly being sent processed. To my surprise crash c23ad887-c371-48d1-85a9-373882091113 - didn't show up immediately in the 3.0 query - I could only find Standard8's test from yesterday evening.

When I asked for crash report c23ad887-c371-48d1-85a9-373882091113, I was greeted with the page telling my that my crash had been archived, and thus didn't appear in the Thunderbird 3.0 query. The Thunderbird QA team needs all report to show, we don't have the same volumes of crash report that Firefox has for now - so we need to be able to have accurate crash numbers, and we need all crashers to show up at least until after a few weeks after 3.0 final is released.

It seems to me that the Thunderbird crashes are throttled, while we had agreed  they wouldn't be.

Comment 2

[Samuel Sidler (old account; do not CC)]

(In reply to comment #1)
> It seems to me that the Thunderbird crashes are throttled, while we had agreed 
> they wouldn't be.

I don't know what this statement means. Who agreed? Doesn't really matter, I suppose.

Thunderbird crashes have never been throttled in the past because they contain "a", "b", or "pre" in their user agent strings. The throttling mechanism we have looks for those and only processes 15% of things that don't have them. (This is actually an issue when we have beta periods for Firefox maintenance releases, but that's a different bug.)

Sounds to me like you want Thunderbird throttling at 100% regardless of version. We may start throttling after you guys release Thunderbird 3, depending on load (maybe 50% or so? I guess we'll see).

Changing the throttling should be a quick fix.

Comment 3

[(not currently active) Ted Mielczarek]

This was implemented in bug 463277.

Comment 4

[Ludovic Hirlimann [:Usul]]

(In reply to comment #2)

> Sounds to me like you want Thunderbird throttling at 100% regardless of
> version. We may start throttling after you guys release Thunderbird 3,

Yes.

> depending on load (maybe 50% or so? I guess we'll see).
> 
> Changing the throttling should be a quick fix.

I'd be veray hapy with that.

Comment 5

[Mark Banner (:standard8)]

(In reply to comment #4)
> (In reply to comment #2)
> 
> > Sounds to me like you want Thunderbird throttling at 100% regardless of
> > version. We may start throttling after you guys release Thunderbird 3,
> 
> Yes.

Just to expand a little here, so apologies in advance if its unnecessary noise:

Up until now, we've been used to 100% of crashes being processed. Typically we get much smaller numbers of users than FF. Therefore whilst we're doing the RCs etc it would be useful to have the same throttling level so that we can get a comparison and good idea of how we're doing compared to the others.

When we release it is quite reasonable to expect extra throttling as there will be a lot more users submitting crashes.

Comment 6

[David Ascher (:davida)]

Throttling on release makes sense if we get a lot of crashes, but we should do that based on real numbers -- it's possible we don't _need_ to throttle, given that the total # of thunderbird users is so much fewer (for now ;-) than FF users, and who knows, maybe we're less crashy too? ;-)

Comment 7

[Wayne Mery (:wsmwk)]

A whole lot of uknowns unfortuantely - except that we know we have far fewer users. And I'd guess after a point release or two it will have substantially settled down.

How quickly can the knob be turned - within an hour, half day, day?

Comment 8

[Ludovic Hirlimann [:Usul]]

Lars could we get traction on this please ?

Comment 9

[K Lars Lohn [:lars] [:klohn]]

sorry, I've been out sick for a week.

Ok, the current configuration already set to give you 100% of Thunderbird crashes. So something must be going wrong with either the rules that exist on the server or there is a bug in the throttler itself.  I am investigating.

Comment 10

[K Lars Lohn [:lars] [:klohn]]

I am unable to reproduce this problem.  I had IT give me the exact collector config file that is in use right now and I installed it in my dev server.  I threw hundreds of Thunderbird crashes at it and 100% we passed on to processing.

_Tsk_, how much time lapsed between your crash submission and your attempt to see it included in a report?  

The results are not generally available immediately.  Sometimes processing can be as far as a couple hours behind.  Unthrottled submissions have to wait in the queue.  If, while waiting for processing, you request a specific crash via the UI, that crash will jump the queue and be processed within the next 60 seconds.  Unfortunately, the recent enhancement to the "waiting for priority processing" is unable to distinguish between the cases where a crash has had its priority raised in the queue or if the crash is being fetched from an archive.  I suspect that the wording of the page lead you to believe that your crash had been archived when it actually had been just prioritized from a wait state.

Comment 11

[Ludovic Hirlimann [:Usul]]

(In reply to comment #10)
> I am unable to reproduce this problem.  I had IT give me the exact collector
> config file that is in use right now and I installed it in my dev server.  I
> threw hundreds of Thunderbird crashes at it and 100% we passed on to
> processing.
> 
> _Tsk_, how much time lapsed between your crash submission and your attempt to
> see it included in a report?  

A few hours. I submitted the crash. Slept my night I would say more than 8. 

Trying to reproduce now. I will have a look at my crash report tomorrow morning.

Comment 12

[Ludovic Hirlimann [:Usul]]

(In reply to comment #11)
> (In reply to comment #10)
> > I am unable to reproduce this problem.  I had IT give me the exact collector
> > config file that is in use right now and I installed it in my dev server.  I
> > threw hundreds of Thunderbird crashes at it and 100% we passed on to
> > processing.
> > 
> > _Tsk_, how much time lapsed between your crash submission and your attempt to
> > see it included in a report?  
> 
> A few hours. I submitted the crash. Slept my night I would say more than 8. 
> 
> Trying to reproduce now. I will have a look at my crash report tomorrow
> morning.
WFM.