1910000 - Airflow task bqetl_artifact_deployment.publish_new_tables failed for exec_date Jul 23 - 25

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Marlene Hirose]

Airflow task bqetl_artifact_deployment.publish_new_tables failed for exec_date Jul 23 - 25

Task link:
https://workflow.telemetry.mozilla.org/dags/bqetl_artifact_deployment/grid?dag_run_id=manual__2024-07-23T15%3A47%3A29.808255%2B00%3A00&task_id=publish_new_tables&tab=logs

Log extract:

[2024-07-23, 19:43:20 UTC] {pod_manager.py:466} INFO - [base] Schema sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_auth_events_v1/schema.yaml updated.
[2024-07-23, 19:43:20 UTC] {pod_manager.py:466} INFO - [base] sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_oauth_events_v1/query.sql ERROR
[2024-07-23, 19:43:20 UTC] {pod_manager.py:466} INFO - [base]  400 POST https://bigquery.googleapis.com/bigquery/v2/projects/moz-fx-data-shared-prod/jobs?prettyPrint=false: moz-fx-fxa-prod-0712:fxa_prod_logs.docker_fxa_oauth_20* does not match any table.
[2024-07-23, 19:43:20 UTC] {pod_manager.py:466} INFO - [base] Location: None
[2024-07-23, 19:43:20 UTC] {pod_manager.py:466} INFO - [base] Job ID: ff5602c0-033d-4b91-8fe7-3bbdc2dea364
[2024-07-23, 19:43:20 UTC] {pod_manager.py:466} INFO - [base] moz-fx-data-shared-prod/firefox_accounts_derived/fxa_oauth_events_v1/query.sql OK
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] Schema sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_oauth_events_v1/schema.yaml updated.
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v2/query.sql OK
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v2/query.sql OK
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] Schema sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v2/schema.yaml updated.
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] sql/moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_stdout_events_v1/query.sql OK
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] moz-fx-data-shared-prod/firefox_accounts_derived/nonprod_fxa_stdout_events_v1/query.sql OK
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] Field stack added to root.jsonPayload.fields
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] Field error added to root.jsonPayload.fields
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] Field k8s_pod_deployment added to root.labels
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] Field errorGroups added to root
[2024-07-23, 19:43:21 UTC] {pod_manager.py:466} INFO - [base] Field root.jsonPayload.fields.stack is missing in schema

Comment 1

[Ben Wu [:benwu]]

Fixed with
https://github.com/mozilla/private-bigquery-etl/pull/438
https://github.com/mozilla/private-bigquery-etl/pull/439
https://github.com/mozilla/private-bigquery-etl/pull/440