1864968 - Airflow task instances are getting deleted due to bug in Airflow 2.6

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Sean Rose [:srose]]

Task instances seem to be getting deleted. When this happens, the task instances are no longer visible the Airflow UI, and it can prevent future DAG runs from running properly if they or their tasks have depends_on_past = True.

We currently think this is due to a known issue in Airflow 2.6 where task instances can get deleted unintentionally. That issue has been fixed in Airflow 2.7.

DAGs affected thus far:

• bqetl_addons
• bqetl_adjust
• bqetl_desktop_funnel
• bqetl_firefox_ios
• bqetl_fog_decision_support
• bqetl_internal_tooling
• bqetl_regrets_reporter_summary
• bqetl_releases
• bqetl_sponsored_tiles_clients_daily
• jetstream
• ltv_daily
• operational_monitoring
• private_bqetl_historical_transactions
• search_alert
• taar_daily
• update_orphaning_dashboard_etl

Dates affected thus far:

• 2023-10-27
• 2023-10-28
• 2023-10-29
• 2023-11-01
• 2023-11-05
• 2023-11-06
• 2023-11-07
• 2023-11-08
• 2023-11-09
• 2023-11-10
• 2023-11-11
• 2023-11-12
• 2023-11-13
• 2023-11-14

Comment 1

[Sean Rose [:srose]]

Some findings after reading through comments on Airflow issue 31179, Airflow PR 32684, and related Airflow issues:

• When manually deleting DAG runs in the Airflow UI the bug will cause it to delete all task instances for the same run ID (i.e. the same timestamp), so this essentially affects all other DAGs on the same schedule as the one you're deleting the DAG run for.
• This might also happen when running airflow db clean? (e.g. Airflow issue 32596)

Comment 2

[Krzysztof Ignasiak [:kik]]

:srose is this issue still valid or should be considered "resolved"? We're at present running Airflow version 2.8.2

Comment 3

[Sean Rose [:srose]]

This was resolved when we upgraded to Airflow 2.7.3 in November.