1257360 - capture metrics on collector throttling

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

The Socorro collector has a LegacyThrottler that has a set of filters for sampling incoming crashes.

This bug covers capturing metrics regarding how many incoming submissions we're rejecting and which filter rule rejected it.


Based on this comment:

https://github.com/mozilla/socorro/pull/3218#issuecomment-197570419

Comment 1

[Peter Bengtsson [:peterbe]]

One brutally simple solution would be to give each throttle rule a name/ID and as it's passing through, not just return True or False (about whether we should process) but also record which name/ID of a throttling rule that allowed it. We can then either...

* just log it and be able to `grep ... | wc -l` to see which one is used, or...

* add the statsd class to the LegacyThrottler class and send a ping to Datadog for each rule used.

Comment 2

[K Lars Lohn [:lars] [:klohn]]

there is an easier way:

grep the collector logs like this:

sudo grep deferring /var/log/messages | grep -v Firefox | grep -v B2G | grep -v Fennec | grep -v Thunderbird

you'll see all the variants of Firefox that people have created including:
    ffgogogo
    WVSecureBrowser
    WASecureBrowser
    Ebon
    BAFI

BTW, that last throttler rule causes these unknown crashes to not go to the processors, we are saving them.  If we want to reject them outright, change the throttle percentage to "None" instead of 0.

Comment 3

[Lonnen :lonnen]

Thanks Lars!

I don't mind them in the raw bucket. Clogging up the processing queue is the risk that concerns me.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm going to grab this one. I'm intrigued by it and I'm going to work on a similar thing. (bug #1293776)

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I implemented this in Antenna. When/if that goes to production, we'll have statsd metrics for which rules match in the Antenna throttler.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Antenna collects throttling rules metrics.

I need to tweak it so that the rule triggered is specified as a tag so then we can do easier graph rollups without having to add each individual rule to the graph.

Still, we have a graph now in the Antenna -prod dashboard. The easy winner is "is_firefox_desktop" which is probably not a surprise to anyone.

Marking as FIXED even though it's done in Antenna which is the rewrite and not the collector.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Gah! I forgot to mark it FIXED. Marking it FIXED for realz!