1393144 - error_aggregates should use submission_date_s3 partitioning

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Datasets: Error Aggregates, enhancement, P1)

Description

[Frank Bertsch [:frank]]

We have largely aligned on partitioning by submission_date_s3, not submission_date. Along with that, the date format should be 'YYYYMMDD', rather than 'YYYY-MM-DD'.

Comment 1

[Frank Bertsch [:frank]]

Old data will need to be moved.

Comment 2

[Arkadiusz Komarzewski [:akomar]]

:frank Batch mode already uses submission_date_s3, whereas streaming one writes submission date.
I think we should standardize this before going live with v2 (as backfilling isn't straightforward now).

However, I don't quite get the meaning of submission_date_s3 in case of a streaming job - after all, data it's processing doesn't go through s3, does it?

Comment 3

[Frank Bertsch [:frank]]

(In reply to akomarzewski from comment #2)
> :frank Batch mode already uses submission_date_s3, whereas streaming one
> writes submission date.
> I think we should standardize this before going live with v2 (as backfilling
> isn't straightforward now).

Yes that should definitely be rectified

> However, I don't quite get the meaning of submission_date_s3 in case of a
> streaming job - after all, data it's processing doesn't go through s3, does
> it?

True. `submission_date_s3` started being our default for submission-date based partitions, mainly because it says something about the partition existing in s3. There's no real formality behind it other than convention.

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-streaming/pull/124

Comment 5

[Arkadiusz Komarzewski [:akomar]]

Will: FYI. I noticed missioncontrol uses submission_date, so this change probably has to be taken into account there.

Comment 6

[William Lachance (:wlach)]

(In reply to akomarzewski from comment #5)
> Will: FYI. I noticed missioncontrol uses submission_date, so this change
> probably has to be taken into account there.

Yup, it's on my radar!