The current mechanism for scheduling tasks on analysis.telemetry.mozilla.org is very simple and as our needs become more complex, we're rapidly outgrowing its capabilities. Specifically, we need the following: - Dependencies between jobs - Improved monitoring (see bug 1232642) - More flexible scheduling options - Support for backfill - Ability to launch one-off jobs
Summary: [meta] Improve scheduling on analysis.telemetry.mozilla.org → [meta] Deploy Airflow
Summary: [meta] Deploy Airflow → [meta] Airflow
I have changed the title as Airflow has been deployed. The remaining work involves slowly moving existing scheduled jobs to Airflow.
Component: Metrics: Pipeline → Scheduling
Product: Cloud Services → Data Platform and Tools
You need to log in before you can comment on or make changes to this bug.