I'm not opposed to a multiple branch-based workflow, but I would like to make sure the additional complexity is warranted, and that it is designed to accommodate a transition to a unified deployment model. Previous discussions around branch deployments were for bug #1609738, which evolved on my recommendation (or obstinance) to not rely on a branch-based workflow. Things that need to be considered: * errors cases * branches per project/realm * rally * syndicate datasets Currently `generated-schemas` is a data source to the ops pipeline. I think in an ideal state, all resources would be deployed by the ops pipeline via data sources, including tables and views. There is obviously some complexity around tables and views, but if `generated-sql` included sufficient metadata for every table, the provisioning of the table could be decoupled from the population of the table via e.g. airflow. The three primary pieces of metadata I see as input for the ops pipeline are: 1. `generated-schemas` 2. [dataset and ACL configuration](https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/tf/prod/envs/prod/bigquery/namespaces.tfvars.json) (also [per-env](https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/tf/prod/envs/prod/bigquery/env_namespaces.tfvars.json)) 1. `generated-sql` I had imagined we would unify these at some point, but for now they exist in three different repos, and the last one isn't a data source to the ops pipeline yet. This metadata deployment model moves the generate/publish-views steps that are currently part of the ops pipeline to be an implementation of the metadata generation pipeline (currently bq-etl CI). It's my understanding that this is currently slow. The central question is whether it is reasonable to rely on `generated-schemas` as "eventually consistent" with production state, for the purposes of the metadata generation pipeline for tables and views. If we unified and updated `generated-schemas`, `generated-sql` and `namespaces.tfvars.json` in lock-step for example, it might be reasonable to treat them collectively as eventually consistent with production (recognizing that there are engineering use cases e.g. CI validation for which this model may be impractical). I think the main issue here is that `generated-sql` is currently being used where `generated-sql-prod` would be used in a system where `generated-sql` is an input to the ops pipeline. I want us to work towards eliminating the publish views steps by making `generated-sql` a data source (where having stable table views that haven't deployed yet is a feature, not a bug), but that's not a short term solution. Here are some short term solutions: 1. Push to `generated-schemas-prod` as mentioned here This requires ops changes and dealing with error conditions. I want to avoid this if possible or to make sure it aligns with my goals above. It may end up being the easiest thing to implement. 1. Parse the commit hash from https://protosaur.dev/mps-deploys/data/mozilla_pipeline_schemas/deploys.json and use that to pull the `generated-schemas` artifact from github Alternatively use "SELECT commit_hash FROM `monitoring-queries`.mozilla_pipeline_schemas.deploys where project = 'moz-fx-data-shared-prod' order by submission_timestamp limit 1" Both of these rely on availability and proper operation of the MPS deployment status dashboard, which I think is probably fine but is also a dependency on non-production infrastructure. Since this JSON is served by GCS on the backend it's very unlikely to be problematic. 1. Parse the commit hash from `schemas_build_id` label metadata I assume the dryrun SA has access to this metadata. This is effectively what the MPS dashboard does on a 15m poll to determine the latest deployment to prod. You would need to pick an arbitrary table to pull this metadata from, but you'd only need one call. This also wouldn't account for schemas deployment errors in the case of a partial deployment. 1. Parse the commit hash from terraform state output This requires ops or permissions changes. I have more thoughts here but I figured it would be better to put this in for discussion before proposing concrete workflows.
Bug 1693353 Comment 1 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
EDIT: per https://github.com/mozilla/bigquery-etl/pull/1814#issuecomment-780885726 the short term solutions here probably aren't necessary. I'm not opposed to a multiple branch-based workflow, but I would like to make sure the additional complexity is warranted, and that it is designed to accommodate a transition to a unified deployment model. Previous discussions around branch deployments were for bug #1609738, which evolved on my recommendation (or obstinance) to not rely on a branch-based workflow. Things that need to be considered: * errors cases * branches per project/realm * rally * syndicate datasets Currently `generated-schemas` is a data source to the ops pipeline. I think in an ideal state, all resources would be deployed by the ops pipeline via data sources, including tables and views. There is obviously some complexity around tables and views, but if `generated-sql` included sufficient metadata for every table, the provisioning of the table could be decoupled from the population of the table via e.g. airflow. The three primary pieces of metadata I see as input for the ops pipeline are: 1. `generated-schemas` 2. [dataset and ACL configuration](https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/tf/prod/envs/prod/bigquery/namespaces.tfvars.json) (also [per-env](https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/tf/prod/envs/prod/bigquery/env_namespaces.tfvars.json)) 1. `generated-sql` I had imagined we would unify these at some point, but for now they exist in three different repos, and the last one isn't a data source to the ops pipeline yet. This metadata deployment model moves the generate/publish-views steps that are currently part of the ops pipeline to be an implementation of the metadata generation pipeline (currently bq-etl CI). It's my understanding that this is currently slow. The central question is whether it is reasonable to rely on `generated-schemas` as "eventually consistent" with production state, for the purposes of the metadata generation pipeline for tables and views. If we unified and updated `generated-schemas`, `generated-sql` and `namespaces.tfvars.json` in lock-step for example, it might be reasonable to treat them collectively as eventually consistent with production (recognizing that there are engineering use cases e.g. CI validation for which this model may be impractical). I think the main issue here is that `generated-sql` is currently being used where `generated-sql-prod` would be used in a system where `generated-sql` is an input to the ops pipeline. I want us to work towards eliminating the publish views steps by making `generated-sql` a data source (where having stable table views that haven't deployed yet is a feature, not a bug), but that's not a short term solution. Here are some short term solutions: 1. Push to `generated-schemas-prod` as mentioned here This requires ops changes and dealing with error conditions. I want to avoid this if possible or to make sure it aligns with my goals above. It may end up being the easiest thing to implement. 1. Parse the commit hash from https://protosaur.dev/mps-deploys/data/mozilla_pipeline_schemas/deploys.json and use that to pull the `generated-schemas` artifact from github Alternatively use "SELECT commit_hash FROM `monitoring-queries`.mozilla_pipeline_schemas.deploys where project = 'moz-fx-data-shared-prod' order by submission_timestamp limit 1" Both of these rely on availability and proper operation of the MPS deployment status dashboard, which I think is probably fine but is also a dependency on non-production infrastructure. Since this JSON is served by GCS on the backend it's very unlikely to be problematic. 1. Parse the commit hash from `schemas_build_id` label metadata I assume the dryrun SA has access to this metadata. This is effectively what the MPS dashboard does on a 15m poll to determine the latest deployment to prod. You would need to pick an arbitrary table to pull this metadata from, but you'd only need one call. This also wouldn't account for schemas deployment errors in the case of a partial deployment. 1. Parse the commit hash from terraform state output This requires ops or permissions changes. I have more thoughts here but I figured it would be better to put this in for discussion before proposing concrete workflows.