1171628 - Alerting on factors that affect pipeline budget

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Katie Parlante]

Including but not limited to:
- avg number of submissions per clientId, avg payload size, number of clients (by channel, day)

Comment 1

[Mark Reid [:mreid]]

Avg submission size on nightly increased by about 8kb for builds after Aug. 20th (I believe due to the addition of USE_COUNTER_* histograms). We should use this as a test case that our alerting is working as expected.

Comment 2

[Mark Reid [:mreid]]

Here is a dashboard to visualize the factors that impact budget:
https://github.com/mozilla/telemetry-budget-dashboard

Alterting to follow.

Comment 3

[Mark Reid [:mreid]]

The above dashboard is now powered by a scheduled report:
https://github.com/mozilla-services/data-pipeline/pull/128

It includes alerting. If any of the actual data exceeds targets, an email will be sent to "[e-mail]". Example log output for yesterday's run:

Checking if we've exceeded targets
ok: Channel nightly, Type main: Actual 25.07GB < Expected 111.76GB (22.4%)
ok: Channel nightly, Type other: Actual 0.04GB < Expected 9.31GB (0.4%)
ok: Channel nightly, Type saved-session: Actual 22.34GB < Expected 89.41GB (25.0%)
ok: Channel release, Type main: Actual 587.42GB < Expected 18626.45GB (3.2%)
ok: Channel release, Type other: Actual 0.97GB < Expected 11641.53GB (0.0%)
ok: Channel release, Type saved-session: Actual 573.37GB < Expected 18626.45GB (3.1%)
ok: Channel aurora, Type main: Actual 52.05GB < Expected 223.52GB (23.3%)
ok: Channel aurora, Type other: Actual 0.03GB < Expected 18.63GB (0.1%)
ok: Channel aurora, Type saved-session: Actual 40.51GB < Expected 178.81GB (22.7%)
ok: Channel Other, Type main: Actual 2.49GB < Expected 1117.59GB (0.2%)
ok: Channel Other, Type other: Actual 0.00GB < Expected 279.40GB (0.0%)
ok: Channel Other, Type saved-session: Actual 1.48GB < Expected 1117.59GB (0.1%)
ok: Channel beta, Type main: Actual 1274.25GB < Expected 4656.61GB (27.4%)
ok: Channel beta, Type other: Actual 0.75GB < Expected 465.66GB (0.2%)
ok: Channel beta, Type saved-session: Actual 1036.32GB < Expected 2793.97GB (37.1%)