1364527 - Search job need to be updated to main_summary_v4

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Thomas Huelbert]

may be other issues Frank is resolving. for tracking.

Comment 1

[Thomas Huelbert]

seems like we need to productionize this - new ticket coming for that

Comment 2

[Frank Bertsch [:frank]]

This job isn't in version control so I can't update the actual version. I have a forked version that I'm using for backfill. I'm starting with 20170508 and will move through to the end of the week. Historically it looks like this job takes ~1 to run one day of data; hopefully then we shouldn't have a problem getting backfill finished by EOD.

Comment 3

[Frank Bertsch [:frank]]

Rollups are completed through 2017-05-11. Let me know if there are anyissues with the data.

Job itself is not updated for v4; leaving this open.

Comment 4

[Frank Bertsch [:frank]]

Looks like Sam completed this, thanks!

Comment 5

[Frank Bertsch [:frank]]

The job is still failing and has not been updated. Sam, I thought you were bumping the version? The notebook is here [0].

[0] https://gist.githubusercontent.com/SamPenrose/856aa21191ef9f0de18c94220cd311a8/raw/afaafc7e5903afb4dde047b43d8e5b2dc2bf6968/daily-main_summary-to-vertica-search-rollups.ipynb

Comment 6

[Sam Penrose]

(In reply to Frank Bertsch [:frank] from comment #5)
> The job is still failing and has not been updated. Sam, I thought you were
> bumping the version? The notebook is here [0].
> 
> [0]
> https://gist.githubusercontent.com/SamPenrose/
> 856aa21191ef9f0de18c94220cd311a8/raw/
> afaafc7e5903afb4dde047b43d8e5b2dc2bf6968/daily-main_summary-to-vertica-
> search-rollups.ipynb

Awesomesauce: when I bumped the version number, GH changed the RAW URI to https://gist.githubusercontent.com/SamPenrose/856aa21191ef9f0de18c94220cd311a8/raw/327af0f490a710eb56dfdd2fceae907f5b5fb14d/daily-main_summary-to-vertica-search-rollups.ipynb . I ran the backlog last night; in the meantime shall we update the airflow job?

Comment 7

[Frank Bertsch [:frank]]

(In reply to Sam Penrose from comment #6)
> (In reply to Frank Bertsch [:frank] from comment #5)
> > The job is still failing and has not been updated. Sam, I thought you were
> > bumping the version? The notebook is here [0].
> > 
> > [0]
> > https://gist.githubusercontent.com/SamPenrose/
> > 856aa21191ef9f0de18c94220cd311a8/raw/
> > afaafc7e5903afb4dde047b43d8e5b2dc2bf6968/daily-main_summary-to-vertica-
> > search-rollups.ipynb
> 
> Awesomesauce: when I bumped the version number, GH changed the RAW URI to
> https://gist.githubusercontent.com/SamPenrose/
> 856aa21191ef9f0de18c94220cd311a8/raw/
> 327af0f490a710eb56dfdd2fceae907f5b5fb14d/daily-main_summary-to-vertica-
> search-rollups.ipynb . I ran the backlog last night; in the meantime shall
> we update the airflow job?

If we're going to update the airflow job, would you mind checking the notebook in to mozilla-reports? Then updating airflow to point to that version. Then at least we have a stable URI while we move the job to mozetl or telemetry-batch-view.

Comment 8

[Frank Bertsch [:frank]]

Anthony did this when he moved the dataset to mozetl. Airflow now points to that repo.