1852753 - Airflow task bqetl_artifact_deployment.publish_new_tables failed for exec_date 2023-09-11

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Krzysztof Ignasiak [:kik]]

Airflow task bqetl_artifact_deployment.publish_new_tables failed for exec_date 2023-09-11

Task link:
https://workflow.telemetry.mozilla.org/dags/bqetl_artifact_deployment/grid?dag_run_id=scheduled__2023-09-11T00%3A00%3A00%2B00%3A00&task_id=publish_new_tables

Log extract:

[2023-09-12, 11:11:10 UTC] {kubernetes_pod.py:870} ERROR - 'NoneType' object has no attribute 'metadata'
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 526, in execute_sync
    self.remote_pod = self.find_pod(self.pod.metadata.namespace, context=context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 473, in find_pod
    raise AirflowException(f"More than one pod running with labels {label_selector}")
airflow.exceptions.AirflowException: More than one pod running with labels dag_id=bqetl_artifact_deployment,kubernetes_pod_operator=True,run_id=scheduled__2023-09-11T0000000000-109912384,task_id=publish_new_tables,already_checked!=True,!airflow-worker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 713, in patch_already_checked
    name=pod.metadata.name,
AttributeError: 'NoneType' object has no attribute 'metadata'
[2023-09-12, 11:11:10 UTC] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 526, in execute_sync
    self.remote_pod = self.find_pod(self.pod.metadata.namespace, context=context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 473, in find_pod
    raise AirflowException(f"More than one pod running with labels {label_selector}")
airflow.exceptions.AirflowException: More than one pod running with labels dag_id=bqetl_artifact_deployment,kubernetes_pod_operator=True,run_id=scheduled__2023-09-11T0000000000-109912384,task_id=publish_new_tables,already_checked!=True,!airflow-worker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py", line 532, in execute
    result = super().execute(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 516, in execute
    return self.execute_sync(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 545, in execute_sync
    self.cleanup(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 671, in cleanup
    raise AirflowException(
airflow.exceptions.AirflowException: Pod publish-new-tables-jz5uur6c returned a failure:

remote_pod: None

Comment 1

[Krzysztof Ignasiak [:kik]]

This seems to have resolved itself.