“value above threshold” warnings for unknown_keys pings
Categories
(Firefox :: Messaging System, task, P2)
Tracking
()
People
(Reporter: emcminn, Unassigned)
References
(Blocks 1 open bug)
Details
Our telemetry monitoring dashboard throws a lot of "value over threshold" errors related to the unknown_keys_volume ping, with the threshold apparently being set to 0. We should set the threshold to some meaningful value, if possible.
See https://github.com/mozilla/metric-hub/blob/main/opmon/firefox-messaging-system.toml
|
||
Comment 1•2 years ago
|
||
ni?Dan to give some extra context.
|
||
Comment 2•2 years ago
|
||
The short version is that the alert threshold is set to fire as soon as a non-zero number of these pings comes in on a given day. In reality, anything below 200 we probably don't care about, because:
- given the recent low end windows release pings per day of 25 millions, 200 is such a tiny number (5/1000 of 1%), the ROI on finding and fixing even a problem of this size is unlikely to be worth it.
- whenever we've dug in, they all seem to just look like random data corruption bugs (eg strings that are similar but not exactly what they should be, with few patterns in the corruption).
@chutten, looking at the above math, even 200 seems like a very low threshold. Are there any rules of thumb in the data world that we might want to apply here?
|
||
Comment 3•2 years ago
|
||
1% is my personal "give a care" threshold (okay, it might have a more... "colloquial" phrasing than that) outside of specific validation exercises where we would expect exact agreement.
|
|
||
Comment 4•2 years ago
|
||
In an ideal world, opmon would allow us to specify this as a percentage of release pings per day, and maybe that's doable. But if it's not quite easy (keeping in mind that iterating on opmon changes takes at least a day per iteration), I'd probably just do the math and set it to, say, 1/2 of 1% in case the number of pings changes materially.
Description
•