1364530 - Migrate Search ETL job to python_mozetl

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Thomas Huelbert]

placeholder, actual steps (version control, monitoring etc) to be added.

Comment 1

[Frank Bertsch [:frank]]

This job would be a good candidate for moving to python_mozetl.

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1364530 - Add daily search rollups via nbconvert #35

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1364530 - Use search rollup from mozetl instead of gist #117

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

I've added pull requests to the corresponding repositories to add a minimally modified version of the search rollup to mozetl and schedule it on airflow. This brings the job into a repository, making changes easier to review and merge.

This job still needs to hit some subset of the following points:

* Backfill all failed dates from May 8 - May 14/15
* Use the execution date passed by airflow in rollup to enable backfill
* Exercise code paths via unit testing
* Remove custom alerting in lieu of airflow alerts, log relevant information to logfiles/stdout

Comment 6

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1364530 - Migrate Search ETL #52

Comment 7

[Sam Penrose]

Shall we close the following bugs as duplicates of this one:

https://bugzilla.mozilla.org/show_bug.cgi?id=1365578
https://bugzilla.mozilla.org/show_bug.cgi?id=1365580