1343875 - Monitor submission latency for pings

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, enhancement, P1)

Description

[Mark Reid [:mreid]]

We should know the distribution of how long it takes for Telemetry data to arrive and be available for analysis.

There is work underway in bug 1336360 to try to send "shutdown" pings immediately, rather than after the next startup, which I expect to have dramatic impact on this latency.

We should monitor the delta between `creationDate` and `Timestamp`, partitioned by normalizedChannel and payload.info.reason as a way to keep an eye on the impact of that work (and because it's a very important aspect of the pipeline in general).

Comment 2

[Mike Trinkala [:trink]]

The initial implementation has proven latency in minutes is too granular as a large portion of the pings are days old.  
The proposal for the alert configuration follows (the actual values will be tuned this sprint):

      latency = {
        hours = 48,  -- 1 - 192  (everything after 192 is added to the last bucket)
        percent = 50,  -- alerts if 50% of submissions are more than 2 days latent latent 
      },

Comment 3

[Mike Trinkala [:trink]]

https://mozilla-services.github.io/lua_sandbox_extensions/moz_telemetry/sandboxes/heka/analysis/moz_telemetry_doctype_monitor.html