Closed Bug 1150030 Opened 6 years ago Closed 6 years ago

TelemetryPing needs to keep track of archived pings

Categories

(Toolkit :: Telemetry, defect)

defect
Not set
normal

Tracking

()

RESOLVED DUPLICATE of bug 1150134
Tracking Status
firefox40 --- affected

People

(Reporter: gfritzsche, Unassigned)

References

Details

For feeding about:healthreport/selfsupport/about:telemetry with data we need to load up to 180 days of archived pings.
We should not hit the disk scanning all pings every session or every time pings are requested.

Instead we should keep track of the archived pings in an index file, say $profile/datareporting/archived/index.
If we don't find this file on delayed init, we should build it by scanning the dir.

The index needs to contain:
* ping type ("main"/"update"/...)
* ping submission date (this also allows removal after 180 days)
* ping uuid (which is also the filename)
What's wrong with hitting the disk, as long as its asynchronous? As long as we store the pings with basic metadata in the filename e.g. YYYY-MM-DD-pingtype-UUID.json then a simple directory index should give us enough information and we can avoid the complexity of keeping a separate index.
Is a directory index cheap enough I/O wise?

I'd also be worried that this doesn't scale and isn't very flexible (we may want to add other lookup-criteria etc.).
Also, without the full date & time i can't do a full ordering of all the pings without reading them from disk.
This would be interesting to e.g. paginate them in order (about:telemetry) or request them one-by-one in order for stitching some data together.
Blocks: 1137252
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> What's wrong with hitting the disk, as long as its asynchronous? As long as
> we store the pings with basic metadata in the filename e.g.
> YYYY-MM-DD-pingtype-UUID.json then a simple directory index should give us
> enough information and we can avoid the complexity of keeping a separate
> index.

That works, we could also do subdirectories named after year-month: https://bugzilla.mozilla.org/show_bug.cgi?id=1137252#c30
Ok, that seems manageable too. I'm still worried that we may need to keep track of other properties, but let's see.
Can anyone comment on the costs of getting the N+1 directory indexes?

We also need to list/index the pending pings, so either we always archive in bug 1137252 or we need to do something similar for pending pings to so we can index them.
This will be handled as part of bug 1150134.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1150134
No longer blocks: 1137252
You need to log in before you can comment on or make changes to this bug.