1689758 - Set table retention period on pioneer-citp-news-disinfo dataset to 3 months

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task, P2)

Description

[Anthony Miyaguchi [:amiyaguchi]]

The retention period of the pioneer-citp-news-disinfo tables should be set to 3 months so we are regularly removing raw data that is no longer necessary. We will first need to confirm that the CITP group has an aggregate pipeline set up before we start to let date partitions expire from the stable tables.

I believe we should be able to set mozPipelineMetadata.expiration_policy.delete_after_days to 90 days in defaults.schema.json

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

:whd, can you confirm that setting the mozPipelineMetadata should set the expiration policy within the pioneer/rally projects as it does on shared-prod? Does it require any intervention on your part?

Comment 2

[Wesley Dawson [:whd]]

(In reply to Anthony Miyaguchi [:amiyaguchi] from comment #1)

:whd, can you confirm that setting the mozPipelineMetadata should set the expiration policy within the pioneer/rally projects as it does on shared-prod?

It should, and this is the correct way to specify retention.

Does it require any intervention on your part?

Yes. By design, any change to retention is considered to be a destructive operation that requires an operator to approve the deployment.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/653

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

I'm going to call this closed, despite the lack of a schema deploy due to recent issues with bug 1690112. I've filed a new bug that will be the retention period going forward in bug 1690637.