1563740 - Modify schema deploy jenkins job for shared-prod and clustering

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, task)

Description

[Jeff Klukas [:klukas] (UTC-4)]

Some changes will be needed to the Jenkins job that deploys BQ schema changes to support the changes accepted in the BigQuery Table Layout and Structure Proposal.

Note that there is a dependency here on an upcoming breaking change to table structure where we will be coercing field names to snake case. The relevant issues are:

• https://github.com/mozilla/gcp-ingestion/issues/671
• https://github.com/mozilla/jsonschema-transpiler/issues/77

There is also the change for clustering support in the BQ sink that should be deployed along with this change to the Jenkins job:

• https://github.com/mozilla/gcp-ingestion/pull/672

The revised Jenkins job should, for each doctype, create or update tables in the following locations:

• shared-[non]prod:<document_namespace>_live.<document_type>_<document_version>
  • PARTITION BY DATE(submission_timestamp)
  • CLUSTER BY submission_timestamp
  • require_partition_filter=TRUE
• shared-[non]prod:<document_namespace>_raw.<document_type>_<document_version>
  • PARTITION BY DATE(submission_timestamp)
  • CLUSTER BY sample_id
  • require_partition_filter=TRUE

My assumption is that we would get the snake casing PRs prepped and staged, the Jenkins job prepped, and then drain the BQ sink Dataflow jobs, merge and deploy all the relevant changes, and then redeploy the BQ sink jobs, pointing at the new live namespaces.

For now, only the live tables will be getting populated and they should have no expiration policy set.

Comment 1

[Jeff Klukas [:klukas] (UTC-4)]

Note that I am out next week on PTO. If this can all come together while I'm out, that's great. If my involvement is needed for coordinating deploy, then I hope we can have as much of this in place as possible to be able to deploy early in the week of 7/15.

Comment 2

[Wesley Dawson [:whd]]

This is implemented in https://github.com/mozilla-services/cloudops-infra/pull/1181 and https://github.com/mozilla-services/cloudops-infra/pull/1183/. I've spun off https://bugzilla.mozilla.org/show_bug.cgi?id=1565074 for the deployment work.