1854148 - Airflow task prerelease_telemetry_aggregates.export_main_avro failed for exec_date 2023-09-20

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Mike Williams [:mwilliams]]

Airflow task prerelease_telemetry_aggregates.export_main_avro failed for exec_date 2023-09-20

Task link:
https://workflow.telemetry.mozilla.org/dags/prerelease_telemetry_aggregates/grid?dag_run_id=scheduled__2023-09-19T00%3A00%3A00%2B00%3A00&task_id=export_main_avro&tab=logs

Log extract:

[2023-09-20, 01:31:16 UTC] {pod_manager.py:187} WARNING - Pod not yet started: export-main-avro-pgbl7y24
[2023-09-20, 01:31:19 UTC] {pod_manager.py:235} INFO - BigQuery error in query operation: Error processing job 'moz-fx-data-airflow-
[2023-09-20, 01:31:19 UTC] {pod_manager.py:235} INFO - gke-prod:bqjob_r11dfa26ce958442d_0000018ab03688ee_1': Access Denied: Table moz-
[2023-09-20, 01:31:19 UTC] {pod_manager.py:235} INFO - fx-data-shared-prod:payload_bytes_decoded.telemetry_telemetry__main_v4: User
[2023-09-20, 01:31:19 UTC] {pod_manager.py:235} INFO - does not have permission to query table moz-fx-data-shared-
[2023-09-20, 01:31:19 UTC] {pod_manager.py:235} INFO - prod:payload_bytes_decoded.telemetry_telemetry__main_v4, or perhaps it does not
[2023-09-20, 01:31:19 UTC] {pod_manager.py:235} INFO - exist in location US.
[2023-09-20, 01:31:20 UTC] {pod_manager.py:260} WARNING - Pod export-main-avro-pgbl7y24 log read interrupted but container base still running
[2023-09-20, 01:31:21 UTC] {pod_manager.py:235} INFO - BigQuery error in query operation: Error processing job 'moz-fx-data-airflow-
[2023-09-20, 01:31:21 UTC] {pod_manager.py:235} INFO - gke-prod:bqjob_r11dfa26ce958442d_0000018ab03688ee_1': Access Denied: Table moz-
[2023-09-20, 01:31:21 UTC] {pod_manager.py:235} INFO - fx-data-shared-prod:payload_bytes_decoded.telemetry_telemetry__main_v4: User
[2023-09-20, 01:31:21 UTC] {pod_manager.py:235} INFO - does not have permission to query table moz-fx-data-shared-
[2023-09-20, 01:31:21 UTC] {pod_manager.py:235} INFO - prod:payload_bytes_decoded.telemetry_telemetry__main_v4, or perhaps it does not
[2023-09-20, 01:31:21 UTC] {pod_manager.py:235} INFO - exist in location US.
[2023-09-20, 01:31:21 UTC] {pod_manager.py:288} INFO - Pod export-main-avro-pgbl7y24 has phase Running
[2023-09-20, 01:31:23 UTC] {kubernetes_pod.py:691} INFO - Skipping deleting pod: export-main-avro-pgbl7y24
[2023-09-20, 01:31:23 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py", line 532, in execute
    result = super().execute(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 516, in execute
    return self.execute_sync(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 545, in execute_sync
    self.cleanup(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 671, in cleanup
    raise AirflowException(
airflow.exceptions.AirflowException: Pod export-main-avro-pgbl7y24 returned a failure:

Looks like a permissions error? From the logs:

BigQuery error in query operation: Error processing job 'moz-fx-data-airflow-gke-prod:bqjob_r11dfa26ce958442d_0000018ab03688ee_1': Access Denied: Table moz-fx-data-shared-prod:payload_bytes_decoded.telemetry_telemetry__main_v4: User does not have permission to query table moz-fx-data-shared-prod:payload_bytes_decoded.telemetry_telemetry__main_v4, or perhaps it does not exist in location US.

Comment 1

[Eduardo Filho [:efilho]]

Looks like someone fixed this already.