Prevent Airflow from running past jobs in dev

RESOLVED FIXED

Status

Data Platform and Tools
Scheduling
P2
normal
RESOLVED FIXED
a year ago
8 months ago

People

(Reporter: sunahsuh, Assigned: sunahsuh)

Tracking

(Blocks: 1 bug)

Details

Attachments

(1 attachment)

(Assignee)

Description

a year ago
Currently, a fresh dev deployment will try to re-run all jobs that are elapsed since the start date, since there's no database record for those runs. This is fine as long as the AWS credentials aren't set in dev. If the credentials happen to be available, however, the clusters for those jobs will actually spin up in aws and this becomes an expensive mistake.

We should figure out how to prevent this from happening and only kick off jobs in dev if they're explicitly triggered.
Created attachment 8801907 [details] [review]
[telemetry-airflow] mozilla:dags-off-by-default > mozilla:master

Updated

8 months ago
Component: Metrics: Pipeline → Scheduling
Product: Cloud Services → Data Platform and Tools

Comment 2

8 months ago
AFAICT this bug was fixed by the referenced PR from :sunahsuh. Further, this setting was made the default by airflow 1.8.0 so our new dev containers should also not exhibit this issue. Closing.
Assignee: nobody → ssuh
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.