Open Bug 1983602 Opened 1 year ago Updated 7 months ago

Investigate and decide what to do about Legacy Telemetry lostEventsCount increases

Categories

(Toolkit :: Telemetry, task, P4)

task

Tracking

()

People

(Reporter: chutten, Unassigned)

References

Details

Legacy Telemetry's events are scheduled on a balance between timeliness and ping size: from the state of having no events, recording an event will trigger a new "event" ping in an hour. If within that hour you happen to record 1000 events, we will instead immediately submit that ping unless a ping has been submitted in the past 10mins. Event storage on the client will never exceed 1000 events, so if more than 1k events have been recorded in that 10min window, every event from 1001 on up will be dropped (not stored, not submitted).

We count how often this happens and report how many events the "event" ping would also have been if it hadn't dropped all of them in payload.lost_events_count.

As far back as data goes (25months), Legacy Telemetry has been dropping ones to tens of millions of events every day. Starting in December of 2024 this number jumped to ~30M per day. And just last Thursday (Aug 14) this number jumped to the hundreds of millions. (query)

Now, this number is easily swayed by bonkers data (and it sometimes is in practice, as can be seen in the results of the query where there are days with truly ludicrous amounts of reported lost events), but I see no reason to not believe the roughly-stable figures of 1s-to-10s of millions until about Fx133 at the end of November 2024, then about 30-40M until August 14. I'm not sure yet whether we should believe the hundreds of millions since Aug 14.

This bug is about figuring out what to do about this.

  • We could do nothing: Legacy Telemetry is going away, and events are likely to be one of the first systems to be turned off. We could merely accept that this is a temporary situation and let it resolve itself with the deprecation and removal phases of the Instrumentation Consolidation Project.
  • We could change behaviour: The "ten minutes" and "one hour" numbers are controlled by prefs. We could change the value of those prefs (decreasing the ten minute throttle to, say, one minute) to allow more events to flow and fewer events to be lost.
  • We could just turn it all off now: that millions of events each day have been discarded due to volume may mean that the value of analyses based on these data are not worth saving. Glean doesn't suffer these problems, and so we can require analyses to use Glean by turning off the faulty data immediately (we can set the maximum and minimum time limit prefs to be days long -- it wouldn't take the number to 0 as days-long sessions could still send 1k events... but the number we'd receive would quickly become a rounding error away from 0).
  • There may be other options, too.

For context, we receive about 3-4B events per day making the 30-40M drop a nearly-perfect 1% decrease.

See Also: → 1982890

The sudden increase starting Aug 14 of dropped events coincides with a sudden increase of high-volume addonsManager.enable.extension and addonsManager.disable.extension events. This is associated with a single addon_id.

The volume has decreased as sharply as it increased, and so I guess it'll get back to its usual volume of lost events in a matter of days.

It did indeed return to normal volumes.

At present there's no penalty to option "Do Nothing", so that's what we've been biasing towards. If we get some time to look at which datasets derive from and who queries telemetry.event and telemetry.events, we would of course prefer option "Just Turn It All Off Now" after working with any consumers to migrate their analysis needs.

Priority: P1 → P2
See Also: → 2030063

Neglected to mention that we've discovered in the intervening months that addons install data derives from telemetry.event and, up until bug 2028765 Remote Settings was using telemetry_derived.events_live. There's also a lot of Experimenter views downstream in there. Gosh, how many experiments were relying on truncated Legacy Event data...

Assignee: chutten → nobody
Status: ASSIGNED → NEW
Priority: P2 → P4
You need to log in before you can comment on or make changes to this bug.