1275889 - Deploy / backfill main_summary v3

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mark Reid [:mreid]]

Now that the code has been refactored and tested, the data needs to be backfilled.

Comment 1

[Mark Reid [:mreid]]

Any new fields should be added before backfilling.

Comment 2

[Mark Reid [:mreid]]

Deployment is done. Backfill still remains.

Comment 3

[Mark Reid [:mreid]]

Backfill will need to take into account the update in bug 1286262 and the outage in bug 1286226

Comment 4

[Sam Penrose]

Do we have an ETA on this?

Comment 5

[GitHub Autolander Bot]

Attachment: [telemetry-airflow] mreid-moz:prepare_main_backfill > mozilla:master

Comment 6

[Mark Reid [:mreid]]

(In reply to Sam Penrose from comment #4)
> Do we have an ETA on this?

I'm planning to kick off the backfill proper once we know that it won't overwhelm the system. The PR above sets it to run a max of 10 days of backfill at once, so that should leave some capacity for all the other jobs.

Once the jobs are all running, I expect it will take a few days to complete.

Comment 7

[GitHub Autolander Bot]

Attachment: [telemetry-airflow] mreid-moz:main_summary_v3_full_backfill_bug1275889 > mozilla:master

Comment 8

[GitHub Autolander Bot]

Attachment: [telemetry-airflow] mreid-moz:backfill_main_summary > mozilla:master

Comment 9

[Mark Reid [:mreid]]

After a fair amount of debugging with Airflow's backfill capability, a backfill-specific version of the DAG is now running, namely the one added in the PR from comment 8.

It will be backfilled for 180 days of history (back to 2016-03-12) per the usual retention period.

Comment 10

[Mark Reid [:mreid]]

Backfill has completed.