1418116 - Backfill main_summary for 20171114 and 20171115

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Bug 1412875 introduced a bug when filtering against the docType in MainSummaryView.

> val filterDocType = conf.docType()
> ...
> Dataset("telemetry").where("docType") { case filterDocType => true } 

The scope of the variable in the case block is local and is the same as `case x => true`, therefore selecting all docType pings. 

This should be fixed with https://github.com/mozilla/telemetry-batch-view/pull/351

20171114 and 20171115 should be backfilled.

Comment 1

[Mark Reid [:mreid]]

20171114 has been backfilled, I re-ran the entire main_summary DAG. There were two task failures:

heavy_users: 
"Exception in thread "main" java.lang.IllegalStateException: Missing previous day's heavy_users data"

hbase_addon_recommender: 
"Invalid number of master nodes found!" and
"'OperationModel' object has no attribute 'deprecated'"


I am restarting heavy_users now, if it fails again I will continue investigating.

Comment 2

[Mark Reid [:mreid]]

Per IRC conversation with Alessio, the hbase_addon_recommender only ever needs the latest data, so we can ignore the failure running against a previous day.

The heavy_users was caused by the way it detects the existence of the previous day's data. Fixed in:
https://github.com/mozilla/telemetry-batch-view/pull/354

Comment 3

[Mark Reid [:mreid]]

20171115 is now running.