Add flag to BQ table deployment script that will only deploy pioneer* tables
Categories
(Data Platform and Tools :: General, task)
Tracking
(Not tracked)
People
(Reporter: bugzilla, Unassigned)
References
Details
Per discussion with whd and Anthony, the simplest thing to do re: pioneer schemas would be continue to use the current repository but add a flag that will only deploy pioneer schemas to:
https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/tf/modules/bigquery/generate_tfjson.sh#L70
We assume that anything that starts with pioneer should be deployed when the flag is set (and the inverse when it's not, e.g. we'll want to make sure the rules skip all schemas that start with "pioneer" instead of skip just the whole word match)
|
||
Comment 1•4 years ago
•
|
||
The inverse logic for this was implemented in shared projects BQ management via https://github.com/mozilla-services/cloudops-infra/pull/2112
The logic for pioneer is significantly refactored to use terraform 0.12, and was written and tested to be sufficiently general to replace the existing 0.11 logic in the shared projects eventually. It uses namespace_match_type (include or exclude) and namespace_match (e.g. ^pioneer-.*$) in order to control the namespaces that a particular ingestion stack includes.
|
Assignee | |
Updated•2 years ago
|
Description
•