1150030 - TelemetryPing needs to keep track of archived pings

Status: RESOLVED DUPLICATE of bug 1150134

(Toolkit :: Telemetry, defect)

Description

[Georg Fritzsche [:gfritzsche]]

For feeding about:healthreport/selfsupport/about:telemetry with data we need to load up to 180 days of archived pings.
We should not hit the disk scanning all pings every session or every time pings are requested.

Instead we should keep track of the archived pings in an index file, say $profile/datareporting/archived/index.
If we don't find this file on delayed init, we should build it by scanning the dir.

The index needs to contain:
* ping type ("main"/"update"/...)
* ping submission date (this also allows removal after 180 days)
* ping uuid (which is also the filename)

Comment 1

[Benjamin Smedberg]

What's wrong with hitting the disk, as long as its asynchronous? As long as we store the pings with basic metadata in the filename e.g. YYYY-MM-DD-pingtype-UUID.json then a simple directory index should give us enough information and we can avoid the complexity of keeping a separate index.

Comment 2

[Georg Fritzsche [:gfritzsche]]

Is a directory index cheap enough I/O wise?

I'd also be worried that this doesn't scale and isn't very flexible (we may want to add other lookup-criteria etc.).

Comment 3

[Georg Fritzsche [:gfritzsche]]

Also, without the full date & time i can't do a full ordering of all the pings without reading them from disk.
This would be interesting to e.g. paginate them in order (about:telemetry) or request them one-by-one in order for stitching some data together.

Comment 4

[Vladan Djeric (:vladan)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> What's wrong with hitting the disk, as long as its asynchronous? As long as
> we store the pings with basic metadata in the filename e.g.
> YYYY-MM-DD-pingtype-UUID.json then a simple directory index should give us
> enough information and we can avoid the complexity of keeping a separate
> index.

That works, we could also do subdirectories named after year-month: https://bugzilla.mozilla.org/show_bug.cgi?id=1137252#c30

Comment 5

[Georg Fritzsche [:gfritzsche]]

Ok, that seems manageable too. I'm still worried that we may need to keep track of other properties, but let's see.
Can anyone comment on the costs of getting the N+1 directory indexes?

We also need to list/index the pending pings, so either we always archive in bug 1137252 or we need to do something similar for pending pings to so we can index them.

Comment 6

[Georg Fritzsche [:gfritzsche]]

This will be handled as part of bug 1150134.