1821789 - Retries required when deploying table that depends on newly added view

Status: NEW

(Data Platform and Tools :: General, defect)

Description

[Anna Scholtz [:ascholtz]]

The order of deployments in bqetl_artifact_deployment is: UDFs, tables, views. In some cases it can happen that a new views gets added, as well as a new query that references this new view. Due to the order of deployments, the table deploy will initially fail, since the view gets created after the table deployments have finished.
The publish_tables Airflow tasks is configured to not fail, even if a table cannot get deployed. So it sometimes takes a day or another manual re-trigger of the deployment tasks for the table to get deployed successfully.

If there are any views that depend on the table, then the publish_views deploy will fail.

We might want to rethink the deployment logic. Either deploy views and tables together in the right order, or ensure the publish_tables task fails so that the Airflow task gets retried automatically.