1295555 - Lower client-side ping archive retention period

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P1)

Description

[Georg Fritzsche [:gfritzsche]]

Currently we archive all Telemetry pings for 180 days on the client.
This has non-zero impact (startup disk I/O, disk fragmentation, ...).

After having this out for a while, we should consider if we still need this or whether we should restrict the retention period (for all or specific ping types).

Comment 1

[Georg Fritzsche [:gfritzsche]]

The archive can serve the following purposes:
(1) power about:healthreport, looking 35 days back [0]
(2) dev & qa mechanism, look up past ping data via about:telemetry etc. - looking back 7-14 days should be sufficient
(3) transparency, offering users insight into past data sent - how far back is this useful?
(4) (planned?) power shield/self-support - are recipes looking at archived ping data still planned? how many days back would they look at archived data? and which ping types?

@bsmedberg: Do you have an opinion on (3)?
@glind: are there plans that would inform (4)?

0: https://github.com/mozilla/fhr-jelly/blob/master/js/data_v4.js#L298

Comment 2

[Benjamin Smedberg]

What is the actual impact? Having more history will be helpful for bug reporting/SHIELD, and matches our server behavior, but if there is obvious impact we can consider lowering it.

Comment 3

[Georg Fritzsche [:gfritzsche]]

The direct impact i know about is:
* side effects like bug 1250357 & FS fragmentation
* (negligible) increased disk use

There will also be background disk activity from scanning/stat()ing the archive files, which i assume we'd like to avoid for e.g. power/battery reasons.

If we are actually planning to use the ping archive for 180 days, then it seems fine.
If we are not planning to use the full archive, it's trivial to lower this (and likewise to increase it again in the future).

Comment 4

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

>> (4) (planned?) power shield/self-support - are recipes looking at archived ping data still planned? how many days back would they look at archived data? and which ping types?

It is not well enough planned to answer yet.  If you have clear reasons to reduce it now, I will not oppose.  I can't make a convincing case for keeping them on the client for 180 days.

Things we know for sure:

- some things will react to current environment (addons installed, prefs set)
- some things should be able to react to recent changes (for repair).  Claim:  recent is << 35 days, so that is fine.
- I have no specific ideas about how to use 'old history'.  I wonder if 'under the last version' is a useful time period or not.  


Summary stats that would be nice to store / access on the client:

- crashes recently (that might be 6 months worth!)
- some summaries of site diversity / page views

Comment 5

[Georg Fritzsche [:gfritzsche]]

From the above, i'm planning to drop the archive retention period to 60 days.
If we need more for specific use cases or pings, i'd be happy to raise this again.

Comment 6

[Georg Fritzsche [:gfritzsche]]

Attachment: Lower client-side ping archive retention period to 60 days

Comment 7

[Georg Fritzsche [:gfritzsche]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=644462ed353ec53cf03904819f6851f086d77466

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/fx-team/rev/089590fd2754
Lower client-side ping archive retention period to 60 days. r=dexter

Comment 9

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/089590fd2754