1880263 - Airflow task bqetl_artifact_deployment .publish_views failed for exec_date 2024-02-14

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Lucia Vargas]

Airflow task bqetl_artifact_deployment .publish_views failed for exec_date 2024-02-14

Task link:
https://workflow.telemetry.mozilla.org/log?dag_id=bqetl_artifact_deployment&task_id=publish_views&execution_date=2024-02-13T00%3A00%3A00%2B00%3A00

Log extract:

google.api_core.exceptions.NotFound: 404 Not found: Dataset moz-fx-data-shared-prod:glean was not found in location US

Comment 1

[Lucia Vargas]

Airflow task bqetl_artifact_deployment .publish_views failed for exec_date 2024-02-15.

Task link:
https://workflow.telemetry.mozilla.org/dags/bqetl_artifact_deployment/grid?dag_run_id=scheduled__2024-02-14T00%3A00%3A00%2B00%3A00&task_id=publish_views&tab=logs

Log extract:

File "/app/bigquery_etl/schema/stable_table_schema.py", line 100, in get_stable_table_schemas"
bq_dataset_family=pipeline_meta["bq_dataset_family"]

KeyError: 'bq_dataset_family'

Comment 2

[Sean Rose [:srose]]

This should have been avoided/fixed by bigquery-etl PR 5037 "Ignore stable table schemas without bq_dataset_family and bq_table metadata" which was merged early yesterday. However, as :akomar mentioned in Slack today the bigquery-etl Docker image hasn't been getting published properly the past few days, so the change from that PR hadn't taken effect in production ETL jobs yet. Now it looks like the issue preventing bigquery-etl Docker images from getting published has been fixed, so the error related to the core Glean schema should go away.

Comment 3

[Lucia Vargas]

After rerunning the teask using the latest image, the task fails with error:

Log extract:

Traceback (most recent call last):
  File "/app/bigquery_etl/cli/view.py", line 225, in publish
    result.append(views_by_id[view_id].publish(target_project, dry_run))
  File "/app/bigquery_etl/view/__init__.py", line 356, in publish
    table.description = self.metadata.description
AttributeError: 'NoneType' object has no attribute 'description'

Comment 4

[Lucia Vargas]

Ran successfully on 2024-02-19.