1821784 - Previously generated files do not get overridden by bqetl_artifact_deployment tasks, ignoring newly added probes

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P3)

Description

[Anna Scholtz [:ascholtz]]

The bigquery-etl docker image the Airflow tasks in bqetl_artifact_deployment use already has the artifacts generated via the generate-sql task. The task itself does run ./bqetl generate all, however generated files that already exist do not get overridden. This was an intentional choice since some views get generated "incorrectly" and instead a manually written version should be used.

In some cases it takes a while until the generated-sql branch gets updated (only happens after a merge to main), so the generated SQL views that this task uses can go out of date, resulting in a deployment failure when a new probe gets added in the meantime.

The current fix is to rerun the CI in bigquery-etl to update the generated SQL, however this is likely not an ideal situation so we might want to come up with a better solution.

Comment 1

[Anna Scholtz [:ascholtz]]

Blocks: [meta] BigQuery Artifact Deployment Improvements

Comment 2

[BMO Github Automation]

Attachment: [mozilla/bigquery-etl] [Bug 1821784] Be explicit about skipping generation for certain Glean artifacts (#4069)