1386639 - Backfill monthly search rollups for 201706 with patch from in-content searches

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Searches for the month of June are inflated due to incontent searches. See related bug 1380002 for the daily backfill.

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Data is backfilled for June and July, under s3://net-mozaws-prod-us-west-2-pipeline-analysis/spenrose/search/to_vertica/monthly/processed-2017-*-01.csv

The naming convention has changed slightly, but I also want to get rid of the manifest files.

Comment 3

[Shraddha Patil [:Shraddha Patil]]

Following up from Anthony's comment-

Matt does that mean we have data in vertica now to be able to fetch via Tableau?

Comment 4

[Matt Pressman [:mpressman]]

Anthony and I have agreed to get rid of the manifest files. They were cumbersome and added complexity to the process. The only thing that's really necessary is to have a consistent naming convention of the files in order to pull down the correct files for the day being processed. I'll need to modify the current process to remove the logic that pulls the manifest to determine which files to download.

Comment 5

[Shraddha Patil [:Shraddha Patil]]

(In reply to Matt Pressman [:mpressman] from comment #4)
> Anthony and I have agreed to get rid of the manifest files. They were
> cumbersome and added complexity to the process. The only thing that's really
> necessary is to have a consistent naming convention of the files in order to
> pull down the correct files for the day being processed. I'll need to modify
> the current process to remove the logic that pulls the manifest to determine
> which files to download.

Thanks Matt for the clarification. This sounds like a sprint work which we can prioritize in our calls.

Comment 6

[Anthony Miyaguchi [:amiyaguchi]]

The bug is not dependent on the state of vertica/tableau, and the data has been backfilled appropriately.

2017-08-22 17:15:12  118578540 processed-2017-06-01.csv
2017-08-22 19:05:04  117934363 processed-2017-07-01.csv