Automatic e-mail notifications of Telemetry validation error rate spikes & drops

RESOLVED FIXED

Status

Webtools
Telemetry Server
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: mreid, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
We should set up automatic alerts based on the validation statistics to be added in bug 972887.
(Reporter)

Updated

3 years ago
Depends on: 972887
No longer depends on: 962811
As discussed yesterday on IRC, I played with the Fourier Transform to see if we can use it to identify a drop in submissions. It seems that it can be used to detect the drop but only after several days, which is not good enough. 

Turns out an ensemble predictor based on Welch's t test of the last 10 daily means works pretty well on the training data I had: https://gist.github.com/vitillo/9023560. Red dots signal a submission drop in the plots.

The predictor takes a series of samples and if enough data is available, it makes a predictions returning True if the submissions are dropping and False otherwise. If not enough data is available, it will just return False; that's the reason you don't see red dots at the beginning of the two plots even though the submissions were dropping (in retrospect though).
(Reporter)

Comment 2

3 years ago
Note: Alerting on the *submission* rate is bug 962811.  This bug is about the *validation error* rate.

Anomaly detection will likely be similar between both bugs, but comment 1 is using data about the submission rate.
See Also: → bug 962811
Sorry about that. If you point me to some data I can have a look at how the algorithm behaves with the validation error rate.
(Reporter)

Comment 4

3 years ago
Validation error rate monitoring has been deployed. Code:
https://github.com/mozilla/telemetry-server/blob/master/monitoring/process_incoming/error_rates.py
(Reporter)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.