Add "Nimbus rollup" metrics event to measure health of experiment recipes in the wild
Categories
(Firefox :: Nimbus Desktop Client, enhancement)
Tracking
()
People
(Reporter: nalexander, Unassigned)
Details
Right now, the Nimbus Desktop Client gives excellent metrics events helping to understand enrollments, unenrollments, validation errors, etc. However, there's very little data helping to understand issues in the wild, including paths not taken. That's sensible: if we have 10s of live experiments enrolling tiny cohorts, it's not helpful to have hundreds of (expensive) events every time we don't enroll. But we could use a middle ground: a single "rollup" event summarizing, perhaps:
- count of experiments witnessed
- count of validation errors
- count of experiments enrolled in, not enrolled in
- count of experiments targeting matched, did not match
Such metrics might have helped diagnose Bug 1793890 a little faster: we might have seen static experiment counts rather than running down issues with data collection, then with targeting, etc.
Presumably this would summarize the events around https://searchfox.org/mozilla-central/rev/f118dae98073bc17efb8604a32abfcb7b4e05880/toolkit/components/nimbus/lib/ExperimentManager.jsm#285.
|
||
Comment 1•1 year ago
|
||
I believe this is covered under https://docs.google.com/document/d/1OHjnBP-MWE5lh5QBkYvChF15Z7iABVIl-cZN8EIywXI/edit and the development is being tracked at https://mozilla-hub.atlassian.net/browse/EXP-3824
Description
•