1656251 - What should we do with unsent data?

Status: NEW

(Data Platform and Tools :: Glean: SDK, enhancement, P4)

Description

[Chris H-C :chutten]

User-lifetime metrics that expire or are removed (see bug 1604854). Ping-lifetime metrics for pings that are never sent. These can result in our users' filesystems being filled with data we never assemble and send and clear.

Is this a concern? Is this a ticking time bomb of ever-increasing size that will bury our users' devices in data?

It might be:

• The default lifetime is "ping", which effectively means "forever" if no ping is submitted
• Metrics can be permanent, or have expiries long enough that they stick around
• Each additional custom ping in send_in_pings multiplies the amount of data to store

But then again it might not be:

• Even Firefox Desktop after 5+ years of Unified Telemetry "only" has 4k metrics in the worst case. (most are use counters). Assuming a ridiculous 1KB of storage requirement per metric, this is only 4MB of data.
  • So as long as we clear expired data from the db, we should be fine. (We do clear it, right? ni?Alessio)

This bug is for deciding whether this is worth

• Instrumenting (and monitoring and alerting)
• Fixing (by, I dunno, expiring old data in the db based on how many months ago it was added or whatever)

Comment 1

[Michael Droettboom [:mdroettboom]]

Related -- maybe we should have telemetry about database size: Bug 1656589

Comment 2

[Michael Droettboom [:mdroettboom]]

The fact that we don't currently support disabling pings is maybe a good thing with respect to this.

Comment 3

[Beatriz Rizental [:brizental]]

We should probably check the size of unsent data before prioritizing this.

Comment 4

[Alessio Placitelli [:Dexter]]

(In reply to Chris H-C :chutten from comment #0)

User-lifetime metrics that expire or are removed (see bug 1604854). Ping-lifetime metrics for pings that are never sent. These can result in our users' filesystems being filled with data we never assemble and send and clear.

Is this a concern? Is this a ticking time bomb of ever-increasing size that will bury our users' devices in data?

It might!

This bug is for deciding whether this is worth

• Instrumenting (and monitoring and alerting)

We are landing instrumentation with bug 1656589. Let's defer any decision after that.