1693353 - Push a branch to mozilla-pipeline-schemas on successful table updates in nonprod and prod

Status: RESOLVED INCOMPLETE

(Data Platform and Tools Graveyard :: Operations, enhancement)

Description

[Jeff Klukas [:klukas] (UTC-4)]

There has been an idea of a generated-schemas-prod branch for a while which would be identical to generated-schemas branch of mozilla-pipeline-schemas, except that the content be updated by the ops logic after a successful terraform apply that updates live and stable tables in BQ.

I discussed recently with :whd and there were some concerns about nuance surrounding the fact that a prod deploy affects not just shared-prod but also mozdata, etc. We deferred the idea, but now it has come back as bigquery-etl CI task time has become bloated due to how much we rely on relatively slow dry runs for checking state in prod. See https://github.com/mozilla/bigquery-etl/pull/1814

My inclination at the moment is that we would ignore view publishing for these branch contents. generated-schemas-prod would be pushed as soon as live and stable tables are updated, but before views publishing.

Perhaps it would make sense to name the branch moz-fx-data-shared-prod to more clearly indicate that it reflects the state of that project only. And we would have a similar branch for nonprod.

I'm looking for additional thoughts on what a precise workflow would look like.

Comment 1

[Wesley Dawson [:whd]]

EDIT: per https://github.com/mozilla/bigquery-etl/pull/1814#issuecomment-780885726 the short term solutions here probably aren't necessary.

I'm not opposed to a multiple branch-based workflow, but I would like to make sure the additional complexity is warranted, and that it is designed to accommodate a transition to a unified deployment model. Previous discussions around branch deployments were for bug #1609738, which evolved on my recommendation (or obstinance) to not rely on a branch-based workflow.

Things that need to be considered:

• errors cases
• branches per project/realm
• rally
• syndicate datasets

Currently generated-schemas is a data source to the ops pipeline. I think in an ideal state, all resources would be deployed by the ops pipeline via data sources, including tables and views. There is obviously some complexity around tables and views, but if generated-sql included sufficient metadata for every table, the provisioning of the table could be decoupled from the population of the table via e.g. airflow.

The three primary pieces of metadata I see as input for the ops pipeline are:

1. generated-schemas
2. dataset and ACL configuration (also per-env)
3. generated-sql

I had imagined we would unify these at some point, but for now they exist in three different repos, and the last one isn't a data source to the ops pipeline yet. This metadata deployment model moves the generate/publish-views steps that are currently part of the ops pipeline to be an implementation of the metadata generation pipeline (currently bq-etl CI). It's my understanding that this is currently slow.

The central question is whether it is reasonable to rely on generated-schemas as "eventually consistent" with production state, for the purposes of the metadata generation pipeline for tables and views. If we unified and updated generated-schemas, generated-sql and namespaces.tfvars.json in lock-step for example, it might be reasonable to treat them collectively as eventually consistent with production (recognizing that there are engineering use cases e.g. CI validation for which this model may be impractical).

I think the main issue here is that generated-sql is currently being used where generated-sql-prod would be used in a system where generated-sql is an input to the ops pipeline. I want us to work towards eliminating the publish views steps by making generated-sql a data source (where having stable table views that haven't deployed yet is a feature, not a bug), but that's not a short term solution.

Here are some short term solutions:

1. Push to generated-schemas-prod as mentioned here
   This requires ops changes and dealing with error conditions. I want to avoid this if possible or to make sure it aligns with my goals above. It may end up being the easiest thing to implement.
2. Parse the commit hash from https://protosaur.dev/mps-deploys/data/mozilla_pipeline_schemas/deploys.json and use that to pull the generated-schemas artifact from github
   Alternatively use "SELECT commit_hash FROM monitoring-queries.mozilla_pipeline_schemas.deploys where project = 'moz-fx-data-shared-prod' order by submission_timestamp limit 1"
   Both of these rely on availability and proper operation of the MPS deployment status dashboard, which I think is probably fine but is also a dependency on non-production infrastructure. Since this JSON is served by GCS on the backend it's very unlikely to be problematic.
3. Parse the commit hash from schemas_build_id label metadata
   I assume the dryrun SA has access to this metadata. This is effectively what the MPS dashboard does on a 15m poll to determine the latest deployment to prod. You would need to pick an arbitrary table to pull this metadata from, but
   you'd only need one call.
   This also wouldn't account for schemas deployment errors in the case of a partial deployment.
4. Parse the commit hash from terraform state output
   This requires ops or permissions changes.

I have more thoughts here but I figured it would be better to put this in for discussion before proposing concrete workflows.

Comment 2

[Daniel Thorn [:relud]]

(In reply to Jeff Klukas [:klukas] (UTC-4) from comment #0)

it has come back as bigquery-etl CI task time has become bloated due to how much we rely on relatively slow dry runs for checking state in prod. See https://github.com/mozilla/bigquery-etl/pull/1814

Another option to consider here is that perhaps generated-sql could be handled in a way that makes it acceptable to generate stable views for tables that have not yet been deployed to prod. For instance we could block view deploys on schema deploys, or we could modify view deploys to issue a warning and continue when a stable table is missing.

Comment 3

[Wesley Dawson [:whd]]

Another option to consider here is that perhaps generated-sql could be handled in a way that makes it acceptable to generate stable views for tables that have not yet been deployed to prod.

This is what I am advocating for in comment #1 in long form. However, I think we're currently using generated-sql as user-facing documentation, which is where the distinction between what's deployed and what's yet to be deployed is relevant.

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

Thank you :whd for all the documentation here. Option (3) actually sounds pretty great. We probably need to update the dry run service to expose labels, but once that's done, I would expect this to be reliable.

Comment 5

[Wesley Dawson [:whd]]

After speaking with :klukas I'm going to WONTFIX this for now since we have multiple options for addressing the immediate use cases and lack of time to begin the more general redesign (which is beyond the scope of this bug).