1398925 - Daily Search Rollup outage 2017-09-01 to 2017-09-06

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Data for the daily search rollup is missing from 2017-09-01 to 2017-09-06.

This is related to a feature/bug added in bug 1365580 that did not work as expected. As a temporary measure, the search rollup was scheduled at UTC+4 on ATMO starting 2017-09-01, which did not give enough time for the `main_summary` job to finish.

This problem was fixed on 2017-09-07.

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

> aws s3 ls s3://net-mozaws-prod-us-west-2-pipeline-analysis/spenrose/search/to_vertica/daily/processed

```
...
2017-09-01 21:24:08          0 processed-2017-09-01.csv
2017-09-02 21:25:34          0 processed-2017-09-02.csv
2017-09-03 21:24:20          0 processed-2017-09-03.csv
2017-09-04 21:24:06          0 processed-2017-09-04.csv
2017-09-05 21:23:48          0 processed-2017-09-05.csv
2017-09-07 12:09:53   49959098 processed-2017-09-06.csv
```

Bug 1397048 introduces a patch that avoids the use of ExternalTaskSensors until a locally tested version of the external dag is created.

Subsequently, the backfill jobs for these dates have been submitted.

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Backill is completed as of 2017-09-11.

```
2017-09-11 14:20:24   47390233 processed-2017-09-01.csv
2017-09-11 14:17:52   44329151 processed-2017-09-02.csv
2017-09-11 14:13:05   43988201 processed-2017-09-03.csv
2017-09-11 14:20:21   49443337 processed-2017-09-04.csv
2017-09-11 14:25:34   50074016 processed-2017-09-05.csv
2017-09-11 14:21:59   49959098 processed-2017-09-06.csv
2017-09-08 12:08:16   49442725 processed-2017-09-07.csv
```

Backfill is necessary from the vertica side.

Comment 3

[Matt Pressman [:mpressman]]

Thanks Anthony!