1113659 - Delete Telemetry data more than 1 year old

Status: RESOLVED FIXED

(Webtools Graveyard :: Telemetry Server, defect)

Description

[Mark Reid [:mreid]]

The Telemetry data bucket in S3 is set to automatically expire data after one year. When we migrated the data to another region in bug 1029616, the timestamps were apparently reset, so we now have data that is older than 1 year (by submission date, rather than by file timestamp) in the published bucket.

We should remove this old data periodically until the auto-expire once again starts taking care of it for us.

Comment 1

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

From where will we be able to access the telemetry data that is more than one year old? We are doing longitudinal studies of the effects of changes to Firefox's TLS configuration, and it would be sad to lose the older aggregated.

Comment 2

[Mark Reid [:mreid]]

There are no plans to delete any aggregate data - this is just to remove the raw telemetry payloads.

If you want to do any analysis/aggregation of the raw data from before December 2013, now is the time (and please let me know so I can hold off the deletion until you're ready).

Comment 3

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Mark Reid [:mreid] from comment #2)
> There are no plans to delete any aggregate data - this is just to remove the
> raw telemetry payloads.

OK. This is good enough for me. I don't have any specific reason for holding up deleting the raw data. Thanks for explaining this!

Comment 4

[Mark Reid [:mreid]]

Attachment: delete_lots_of_things.py

Here is a utility for quickly deleting a list of keys from a bucket.  Beware, this *will* delete data, so be very careful.

Comment 5

[Mark Reid [:mreid]]

I used the above utility to delete the old data from Sept. 2013 to Jan 22, 2014.  

I also put a list of files by day into S3 at s3://mreid-test-src/telemetry-published-v2.s3_filenames.20150121.tar.gz to make this easy next time around.

We should periodically purge data until the bucket's auto-expiry policy starts to take effect.  That should start happening in late July, 2015.