Deploy / backfill main_summary v3

RESOLVED FIXED

Status

Cloud Services
Metrics: Pipeline
P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: mreid, Assigned: mreid)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

(Assignee)

Description

2 years ago
Now that the code has been refactored and tested, the data needs to be backfilled.
(Assignee)

Comment 1

2 years ago
Any new fields should be added before backfilling.
Assignee: nobody → mreid
Points: --- → 2
Priority: -- → P1
(Assignee)

Updated

2 years ago
Depends on: 1270482, 1270505, 1265874
(Assignee)

Updated

2 years ago
Depends on: 1282758
(Assignee)

Updated

2 years ago
Depends on: 1274981
(Assignee)

Comment 2

2 years ago
Deployment is done. Backfill still remains.
(Assignee)

Comment 3

2 years ago
Backfill will need to take into account the update in bug 1286262 and the outage in bug 1286226
Blocks: 1286226
(Assignee)

Updated

2 years ago
Depends on: 1287585

Comment 4

2 years ago
Do we have an ETA on this?
Flags: needinfo?(mreid)
Created attachment 8775127 [details] [review]
[telemetry-airflow] mreid-moz:prepare_main_backfill > mozilla:master
(Assignee)

Comment 6

2 years ago
(In reply to Sam Penrose from comment #4)
> Do we have an ETA on this?

I'm planning to kick off the backfill proper once we know that it won't overwhelm the system. The PR above sets it to run a max of 10 days of backfill at once, so that should leave some capacity for all the other jobs.

Once the jobs are all running, I expect it will take a few days to complete.
Flags: needinfo?(mreid)
(Assignee)

Updated

2 years ago
Attachment #8775127 - Flags: review?(rvitillo)
Attachment #8775127 - Flags: review?(rvitillo) → review+
Created attachment 8788436 [details] [review]
[telemetry-airflow] mreid-moz:main_summary_v3_full_backfill_bug1275889 > mozilla:master
Created attachment 8789769 [details] [review]
[telemetry-airflow] mreid-moz:backfill_main_summary > mozilla:master
(Assignee)

Comment 9

2 years ago
After a fair amount of debugging with Airflow's backfill capability, a backfill-specific version of the DAG is now running, namely the one added in the PR from comment 8.

It will be backfilled for 180 days of history (back to 2016-03-12) per the usual retention period.
(Assignee)

Comment 10

2 years ago
Backfill has completed.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.