Closed Bug 1354725 Opened 9 years ago Closed 7 years ago

Standardize on submission_date_s3 dimension for direct-to-parquet outputs

Categories

(Data Platform and Tools :: General, enhancement, P3)

Product:
Data Platform and Tools
Component:
General
Type:
enhancement
Points:
1

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: whd, Unassigned)

References

Details

(Whiteboard: [DataOps])

While working on bug #1353784 I noticed that we have already had a convention for submission_date as an s3 partition from telemetry-batch-view, which is submission_s3_date. We should change the d2p outputs to use this as the partition name instead of submission. This would require checking for and possibly rewriting scheduled queries that are using submission currently. However, I believe most d2p are not being widely used (yet).
Component: Metrics: Pipeline → Pipeline Ingestion
Product: Cloud Services → Data Platform and Tools
See Also: → 1422952
Whiteboard: [SvcOps] → [DataOps]
Priority: P2 → P3
This would require changing a bunch of queries and changing where a bunch of data is written. With an infrastructure migration impending, we've decided not to bother with standardizing on the current infra (but we will be standardizing on the migrated infra).
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Component: Pipeline Ingestion → General
You need to log in before you can comment on or make changes to this bug.