Alerting on factors that affect pipeline budget

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: kparlante, Assigned: mreid)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [unifiedTelemetry][rC])

(Reporter)

Description

3 years ago
Including but not limited to:
- avg number of submissions per clientId, avg payload size, number of clients (by channel, day)
(Assignee)

Updated

3 years ago
Priority: -- → P2
(Assignee)

Updated

3 years ago
Whiteboard: [unifiedTelemetry][rC]
(Reporter)

Updated

3 years ago
Blocks: 1182693

Updated

3 years ago
Iteration: --- → 43.1 - Aug 24
(Assignee)

Comment 1

3 years ago
Avg submission size on nightly increased by about 8kb for builds after Aug. 20th (I believe due to the addition of USE_COUNTER_* histograms). We should use this as a test case that our alerting is working as expected.
(Assignee)

Comment 2

3 years ago
Here is a dashboard to visualize the factors that impact budget:
https://github.com/mozilla/telemetry-budget-dashboard

Alterting to follow.
(Assignee)

Comment 3

3 years ago
The above dashboard is now powered by a scheduled report:
https://github.com/mozilla-services/data-pipeline/pull/128

It includes alerting. If any of the actual data exceeds targets, an email will be sent to "telemetry-alerts@mozilla.com". Example log output for yesterday's run:

Checking if we've exceeded targets
ok: Channel nightly, Type main: Actual 25.07GB < Expected 111.76GB (22.4%)
ok: Channel nightly, Type other: Actual 0.04GB < Expected 9.31GB (0.4%)
ok: Channel nightly, Type saved-session: Actual 22.34GB < Expected 89.41GB (25.0%)
ok: Channel release, Type main: Actual 587.42GB < Expected 18626.45GB (3.2%)
ok: Channel release, Type other: Actual 0.97GB < Expected 11641.53GB (0.0%)
ok: Channel release, Type saved-session: Actual 573.37GB < Expected 18626.45GB (3.1%)
ok: Channel aurora, Type main: Actual 52.05GB < Expected 223.52GB (23.3%)
ok: Channel aurora, Type other: Actual 0.03GB < Expected 18.63GB (0.1%)
ok: Channel aurora, Type saved-session: Actual 40.51GB < Expected 178.81GB (22.7%)
ok: Channel Other, Type main: Actual 2.49GB < Expected 1117.59GB (0.2%)
ok: Channel Other, Type other: Actual 0.00GB < Expected 279.40GB (0.0%)
ok: Channel Other, Type saved-session: Actual 1.48GB < Expected 1117.59GB (0.1%)
ok: Channel beta, Type main: Actual 1274.25GB < Expected 4656.61GB (27.4%)
ok: Channel beta, Type other: Actual 0.75GB < Expected 465.66GB (0.2%)
ok: Channel beta, Type saved-session: Actual 1036.32GB < Expected 2793.97GB (37.1%)
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.