1285621 - Number of pings per day is falling

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mauro Doglio [:mdoglio]]

Attachment: ping_plot.png

I found out that there is a suspicious fall in the number of pings ingested by telemetry per day. The anomaly seems to start on July 4th where the chart stops following the same pattern as the previous weeks. It would be great if someone could have a look at out infra to make sure everything is working as expected in the ping ingestion.

Comment 1

[Mauro Doglio [:mdoglio]]

Attachment: ping_ingestion_fall.ipynb

Here is the notebook I used to plot the data

Comment 2

[Blake Imsland [:robotblake]]

After doing some research with :relud and :whd we've found the issue and will be sending out an e-mail with information and next steps. The gist is that our monitoring missed a data loader issue and we'll need to do some backfill for the affected days.

Comment 3

[Thomas Huelbert]

this might be resolved, please confrim

Comment 4

[Mark Reid [:mreid]]

The underlying error has long been corrected, there may still be some lingering backfill issues, but let's deal with those in bug 1286226.