962811 - Automatic e-mail notifications of Telemetry submission rate spikes & drops

Status: RESOLVED FIXED

(Webtools Graveyard :: Telemetry Server, defect)

Description

[Vladan Djeric (:vladan)]

We should set up automatic e-mail notifications for significant changes in Telemetry submission rates. 

[e-mail] should probably be on the recipient list

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

CC'ing trink, as we already have these measures in heka, so I'm hoping this is just a small heka hack :)

Comment 2

[Mark Reid [:mreid]]

See also: https://github.com/mozilla-services/heka/issues/677

Comment 3

[Mark Reid [:mreid]]

In addition to the Standard-Deviation-based alerting implemented by :trink, I've added a cron job to monitor the submission rates externally using :rvitillo's suggestion of a predictor based on Mann-Whitney's U test from https://gist.github.com/vitillo/9023560/.

It was added to the telemetry-server project in this commit: https://github.com/mozilla/telemetry-server/commit/abc644b0b70777889b8c29d45f05fb2eae69b302

Comment 4

[Mike Trinkala [:trink]]

Any idea what causes the 4am PST interruption to the data stream every Saturday (consistently generationg the 4:05 alert)?

Comment 5

[Mark Reid [:mreid]]

I think it's just due to Saturdays being consistently ~15-20% lower volume than Fridays, so it hits the normal stddev cutoff...

Comment 6

[Roberto Agostino Vitillo (:rvitillo)]

Mike, what are you using as reference distribution for your stddev approach?

Comment 7

[Mike Trinkala [:trink]]

The data as-is will not cause an alert.  There must have been an interruption in the data stream and when it was over the old data was backfilled correcting the graph.

Comment 8

[Mike Trinkala [:trink]]

(In reply to Mike Trinkala [:trink] from comment #7)
> The data as-is will not cause an alert.  There must have been an
> interruption in the data stream and when it was over the old data was
> backfilled correcting the graph.

Ignore the comment above... It helps if I look at the right graph (it just clips the threshold)

http://ec2-50-112-66-71.us-west-2.compute.amazonaws.com:4352/alert_threshold.html?win=15&col=1&sd=1.5&file=TelemetryChannelMetrics60DaysAggregatorAlerting.ALL.cbuf

Comment 9

[Mike Trinkala [:trink]]

All this is in and has been running on the Ops supported shared Heka we just need to get the telemetry edge nodes updated to send the data there. https://heka.shared.us-west-2.prod.mozaws.net/ (you will probably need to ask whd for access to the dashboard)

Comment 10

[Mark Reid [:mreid]]

The automated alerting has been running for a long time.