Closed Bug 742496 Opened 12 years ago Closed 9 years ago

Telemetry data retention policy

Categories

(Mozilla Metrics :: Data/Backend Reports, defect)

Product:
Mozilla Metrics
Component:
Data/Backend Reports
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Moved to JIRA

People

(Reporter: lmandel, Unassigned)

References

Details

(Whiteboard: [JIRA METRICS-790] [Telemetry:P1]) 

Telemetry does not currently have a data retention policy. I would like to discuss the requirements and develop a policy so that it is clear to users how long Telemetry data will be retained. (Note that I consider indefinitely to be a valid request so long as it comes with justification.)

To assist in our decision we should identify:

1. any physical constraints (I'm thinking disk space).
Daniel - Can you please try to determine how much disk space is currently required to store the Telemetry data from a single day?

2. usage patterns for the data to determine over what time period data is useful.

We should also consider whether having different policies for the different channels is warranted as well.
Marking: in group of > 33 asks for Telemetry that need PM priority before triage/scheduling.
Status: NEW → ASSIGNED
Whiteboard: [Telemetry] → Telemetry -- needs PM project priority
Triaged.
Target Milestone: Unreviewed → Backlogged - BZ
> 1. any physical constraints (I'm thinking disk space).
> Daniel - Can you please try to determine how much disk space is currently
> required to store the Telemetry data from a single day?
> 

At current data size (per record):

192 GB (= 64GB (compressed) x 3 copies (for Hadoop replication)) per day

Please note that the data size per record has doubled from ~17k in Q1 to ~35k (now)

--
https://metrics.mozilla.com/projects/browse/METRICS-790
Target Milestone: Backlogged - BZ → Targeted - JIRA
Depends on: 668037
Whiteboard: Telemetry -- needs PM project priority → [Telemetry:P1]
Whiteboard: [Telemetry:P1] → [JIRA METRICS-790] [Telemetry:P1]
Do you still want this information? I'm not sure if you've already set a lifecycle for how long we keep telemetry data.
Flags: needinfo?(lmandel)
bsmedberg - You own Telemetry now. Do you already have a policy for data retention? Is there a need to keep this bug open?
Flags: needinfo?(lmandel) → needinfo?(benjamin)
This is now folded into the Firefox data retention policy in general, along with FHR. Officially we're allowed to do 13-month retention, but we currently plan on storing 180 days.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Flags: needinfo?(benjamin)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.