1344234 - Add hindsight monitor for ping volumes

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Georg Fritzsche [:gfritzsche]]

We want:
- email alerts to [e-mail]
- graphing over time, per-channel, per-type
- regression detection, per-channel, per-type

This should give us timely alerts of any regressions in ping volumes, e.g. when a ping is suddenly sent much more often or stops being sent.

Comment 1

[Mike Trinkala [:trink]]

The proposal for the alert configuration follows.  However, there are several issues.

      volume = {
        inactivity_timeout = 5, -- minutes with no data
        percent_delta = 10, -- optional
      },

Issues:
1) the percent_delta can be based on a model
- The model requires much more setup and since volumes can be highly variable much more maintenance and constant tuning.

2) the percent_delta can be based the same day of the previous week.
- Using the previous week's data is easy but will be noisy.  e.g. if there is an anomaly this week (infrastructure deploy) the system will alert now and a week from now.
- the percent_delta can creep up each week so in less than two months the volume could have doubled without an alert

Comment 3

[Mike Trinkala [:trink]]

https://mozilla-services.github.io/lua_sandbox_extensions/moz_telemetry/sandboxes/heka/analysis/moz_telemetry_doctype_monitor.html