1854147 - Airflow task mobile_aggregates.export_mobile_metrics_avro failed for exec_date 2023-09-20

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, defect)

Description

[Mike Williams [:mwilliams]]

Airflow task mobile_aggregates.export_mobile_metrics_avro failed for exec_date 2023-09-20

Task link:
https://workflow.telemetry.mozilla.org/dags/mobile_aggregates/grid?dag_run_id=scheduled__2023-09-19T00%3A00%3A00%2B00%3A00&task_id=export_mobile_metrics_avro&tab=logs

Log extract:

[2023-09-20, 01:32:29 UTC] {pod_manager.py:235} INFO - BigQuery error in query operation: Error processing job 'moz-fx-data-airflow-
[2023-09-20, 01:32:29 UTC] {pod_manager.py:235} INFO - gke-prod:bqjob_r689e39013dceaf4a_0000018ab0379ae2_1': Access Denied: Table moz-
[2023-09-20, 01:32:29 UTC] {pod_manager.py:235} INFO - fx-data-shared-
[2023-09-20, 01:32:29 UTC] {pod_manager.py:235} INFO - prod:payload_bytes_decoded.telemetry_telemetry__mobile_metrics_v1: User does not
[2023-09-20, 01:32:29 UTC] {pod_manager.py:235} INFO - have permission to query table moz-fx-data-shared-
[2023-09-20, 01:32:29 UTC] {pod_manager.py:235} INFO - prod:payload_bytes_decoded.telemetry_telemetry__mobile_metrics_v1, or perhaps it
[2023-09-20, 01:32:29 UTC] {pod_manager.py:235} INFO - does not exist in location US.
[2023-09-20, 01:32:30 UTC] {pod_manager.py:288} INFO - Pod export-mobile-metrics-avro-595vsgqq has phase Running
[2023-09-20, 01:32:32 UTC] {kubernetes_pod.py:691} INFO - Skipping deleting pod: export-mobile-metrics-avro-595vsgqq
[2023-09-20, 01:32:32 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py", line 532, in execute
    result = super().execute(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 516, in execute
    return self.execute_sync(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 545, in execute_sync
    self.cleanup(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 671, in cleanup
    raise AirflowException(
airflow.exceptions.AirflowException: Pod export-mobile-metrics-avro-595vsgqq returned a failure:

Looks like a permissions error?

Comment 1

[Mike Williams [:mwilliams]]

Failed again today 2023-09-21 -- looks like the same error/logs

Comment 2

[anicholson]

This dag is paused and hasn't run since 2023-09-22. The last run was successful as well, I think it's fine to mark this resolved