1472710 - Remove old inactive dags from airflow dag list UI

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[bugzilla]

We have a couple old dags that have been removed from the codebase but show up in the dag list with the tooltip "This DAG isn't available in the web server's DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database."

We should figure out how to clean up after these old dags, especially as the list of dags grows beyond one page.

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

It looks like all the references to the DAG from the metadata table.

https://issues.apache.org/jira/browse/AIRFLOW-1002

It looks like some new functionality was added in airflow 1.10 to do this through a REST interface, otherwise it needs to be done manually. I don't think I have access to the production postgres server where the airflow_db lives, but I can write the script that needs to be run.

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: sweep_airflow_dags.py

I don't have access to the Airflow cluster. :hwoo can you run this on the web node after checking that the script looks sane? This can be run in any container that has access to the postgres database.

Comment 3

[Harold Woo]

Script isn't working.  Unsure how the connection is created, would require a dive into the source code.

The other comments on the stackoverflow page you referenced seem to indicate that these metadata relations exist but I cannot find them in the postgres db when looking around.

Comment 4

[Harold Woo]

Ok figured it out.  For future reference:

The postgres metadata db is separate and defined by AIRFLOW_DATABASE_URL.  I had to create a connection via the airflow UI at Admin -> Connections called 'airflow_db_meta' to reference in the script.

hook = PostgresHook(postgres_conn_id = "airflow_db_meta")

Then run the script on the web container in a python shell.