Closed Bug 1872187 Opened 2 years ago Closed 2 years ago

Airflow task glam.clients_histogram_aggregates failed for exec_date 2023-12-23

Categories

(Data Platform and Tools :: General, defect)

Product:
Data Platform and Tools
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: curtis, Assigned: curtis)

Details

(Whiteboard: [airflow-triage])

Airflow task glam.clients_histogram_aggregates failed for exec_date 2023-12-23

Task link:
https://workflow.telemetry.mozilla.org/dags/glam/grid?execution_date=2023-12-23T08%3A00%3A00%2B00%3A00&dag_run_id=scheduled__2023-12-23T02%3A00%3A00%2B00%3A00&task_id=clients_histogram_aggregates

Downstream tasks may need to be manually rerun after this task is fixed:
bqetl_glam_export.wait_for_extracts https://workflow.telemetry.mozilla.org/dags/bqetl_glam_export/grid?dag_run_id=scheduled__2023-12-23T08%3A00%3A00%2B00%3A00&task_id=wait_for_extracts
glam_glean_imports.wait_for_glam https://workflow.telemetry.mozilla.org/dags/glam_glean_imports/grid?dag_run_id=scheduled__2023-12-23T05%3A00%3A00%2B00%3A00&task_id=wait_for_glam

Log extract:

[2023-12-24, 05:46:18 UTC] {subdag.py:187} INFO - Execution finished. State is failed
[2023-12-24, 05:46:18 UTC] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1518, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode, session=session)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1684, in _execute_task_with_callbacks
    self.task.post_execute(context=context, result=result)  # type: ignore[union-attr]
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/subdag.py", line 190, in post_execute
    raise AirflowException(f"Expected state: SUCCESS. Actual state: {dag_run.state}")
airflow.exceptions.AirflowException: Expected state: SUCCESS. Actual state: failed
[2023-12-24, 05:46:18 UTC] {taskinstance.py:1400} INFO - Marking task as FAILED. dag_id=glam, task_id=clients_histogram_aggregates, execution_date=20231223T020000, start_date=20231224T052517, end_date=20231224T054618
[2023-12-24, 05:46:18 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.10/site-packages/airflow/utils/email.py:154: RemovedInAirflow3Warning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
  send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
[2023-12-24, 05:46:18 UTC] {email.py:270} INFO - Email alerting: attempt 1
[2023-12-24, 05:46:18 UTC] {email.py:281} INFO - Sent an alert email to ['telemetry-alerts@mozilla.com', 'akomarzewski@mozilla.com', 'efilho@mozilla.com', 'linhnguyen@mozilla.com']
[2023-12-24, 05:46:19 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 2370120 for task clients_histogram_aggregates (Expected state: SUCCESS. Actual state: failed; 499789)

This task is fixed now; rerunning seemed to be enough to fix it. I just need to go through and rerun the downstream tasks and I can close this out.

Downstream tasks have all run successfully. Closing this out

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.