1844881 - Airflow task graphics_telemetry .graphics_dashboard failed for exec_date 2023-07-21

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Winnie Chan [:winnie]]

Airflow task graphics_telemetry .graphics_dashboard failed for exec_date 2023-07-21

Task link:
https://prod.telemetry-airflow.prod.dataservices.mozgcp.net/dags/graphics_telemetry/grid?dag_run_id=scheduled__2023-07-20T03%3A00%3A00%2B00%3A00&task_id=graphics_dashboard

Log extract:

gha-worker-b557d65b8-zlx24
*** Found remote logs:
***   * gs://airflow-remote-logs-prod-prod/dag_id=graphics_telemetry/run_id=scheduled__2023-07-20T03:00:00+00:00/task_id=graphics_dashboard/attempt=6.log
[2023-07-22, 00:09:26 UTC] {taskinstance.py:1090} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: graphics_telemetry.graphics_dashboard scheduled__2023-07-20T03:00:00+00:00 [queued]>
[2023-07-22, 00:09:26 UTC] {taskinstance.py:1090} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: graphics_telemetry.graphics_dashboard scheduled__2023-07-20T03:00:00+00:00 [queued]>
[2023-07-22, 00:09:26 UTC] {taskinstance.py:1288} INFO - 
--------------------------------------------------------------------------------
[2023-07-22, 00:09:26 UTC] {taskinstance.py:1289} INFO - Starting attempt 6 of 6
[2023-07-22, 00:09:26 UTC] {taskinstance.py:1290} INFO - 
--------------------------------------------------------------------------------
[2023-07-22, 00:09:27 UTC] {taskinstance.py:1309} INFO - Executing <Task(SubDagOperator): graphics_dashboard> on 2023-07-20 03:00:00+00:00
[2023-07-22, 00:09:27 UTC] {standard_task_runner.py:55} INFO - Started process 23581 to run task
[2023-07-22, 00:09:27 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'graphics_telemetry', 'graphics_dashboard', 'scheduled__2023-07-20T03:00:00+00:00', '--job-id', '282822', '--raw', '--subdir', 'DAGS_FOLDER/graphics_telemetry.py', '--cfg-path', '/tmp/tmpzjrsaygg']
[2023-07-22, 00:09:27 UTC] {standard_task_runner.py:83} INFO - Job 282822: Subtask graphics_dashboard
[2023-07-22, 00:09:27 UTC] {task_command.py:389} INFO - Running <TaskInstance: graphics_telemetry.graphics_dashboard scheduled__2023-07-20T03:00:00+00:00 [running]> on host gha-worker-b557d65b8-zlx24
[2023-07-22, 00:09:27 UTC] {taskinstance.py:1516} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=[e-mail],[e-mail]
AIRFLOW_CTX_DAG_OWNER=[e-mail]
AIRFLOW_CTX_DAG_ID=graphics_telemetry
AIRFLOW_CTX_TASK_ID=graphics_dashboard
AIRFLOW_CTX_EXECUTION_DATE=2023-07-20T03:00:00+00:00
AIRFLOW_CTX_TRY_NUMBER=6
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2023-07-20T03:00:00+00:00
[2023-07-22, 00:09:27 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.10/site-packages/airflow/utils/context.py:205: AirflowContextDeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_start' or 'logical_date' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2023-07-22, 00:09:27 UTC] {subdag.py:173} INFO - Found existing DagRun: scheduled__2023-07-20T03:00:00+00:00
[2023-07-22, 00:10:27 UTC] {base.py:228} INFO - Success criteria met. Exiting.
[2023-07-22, 00:10:27 UTC] {subdag.py:186} INFO - Execution finished. State is failed
[2023-07-22, 00:10:27 UTC] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1383, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1531, in _execute_task_with_callbacks
    self.task.post_execute(context=context, result=result)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/subdag.py", line 189, in post_execute
    raise AirflowException(f"Expected state: SUCCESS. Actual state: {dag_run.state}")
airflow.exceptions.AirflowException: Expected state: SUCCESS. Actual state: failed
[2023-07-22, 00:10:27 UTC] {taskinstance.py:1327} INFO - Marking task as FAILED. dag_id=graphics_telemetry, task_id=graphics_dashboard, execution_date=20230720T030000, start_date=20230722T000926, end_date=20230722T001027
[2023-07-22, 00:10:27 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.10/site-packages/airflow/utils/email.py:152: RemovedInAirflow3Warning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
  send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)

[2023-07-22, 00:10:27 UTC] {email.py:268} INFO - Email alerting: attempt 1
[2023-07-22, 00:10:27 UTC] {email.py:280} INFO - Sent an alert email to ['[e-mail]', '[e-mail]']
[2023-07-22, 00:10:27 UTC] {standard_task_runner.py:100} ERROR - Failed to execute job 282822 for task graphics_dashboard (Expected state: SUCCESS. Actual state: failed; 23581)
[2023-07-22, 00:10:27 UTC] {local_task_job.py:212} INFO - Task exited with return code 1

Comment 1

[Krzysztof Ignasiak [:kik]]

It appears re-running the failed subdags worked.