telemetry-aggregator job should depend on past

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rvitillo, Unassigned)

Tracking

(Blocks: 1 bug)

Details

User Story

By setting depend_on_past = True in the job's configuration for Airflow we ensure that only a single job at the time will be run during backfill, which is what we want to avoid DDoSiNg the database.
Comment hidden (empty)

Updated

2 years ago
Points: --- → 1
Priority: -- → P2
(Reporter)

Updated

2 years ago
Component: Metrics: Pipeline → Datasets: Telemetry Aggregates
Product: Cloud Services → Data Platform and Tools
The airflow job is already setup using depend_on_past [0]. Thought we can possibly consider removing it since we now have a read-replica; heavy writing won't affect users anymore.

[0] https://github.com/mozilla/telemetry-airflow/blob/master/dags/telemetry_aggregates.py#L7
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.