1380002 - Backfill daily search rollups from 20170601 with patch from in-content searches

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Search rollups were affected by a regression that removed the search_counts.source whitelist filtering. Without this whitelist, many searches are double counted. A recent PR fixed this issue in both search_rollups and topline_summary. [1]

The daily_search_rollups dataset should be backfilled from 20170601 onward. 

[1] https://github.com/mozilla/python_mozetl/commit/149755cb93c6d1917bab60084bb3240fae537341

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

What is necessary to backfill the data on the IT side? The current backfill process rewrites the existing csv data with the corrected values.

Comment 2

[Matt Pressman [:mpressman]]

Thanks for the backfill. I've cleaned up the existing June data in our data warehouse and have successfully loaded it. I think I'm all good!

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

This data set was backfilled on the data platform side July 12th, affecting data from June 1st to July 12th. Data was loaded into the final data store in comment #2.

Comment 4

[arana]

The data for June,2017 looks incorrect. The total search count is double than each of the previous and following months. Is there a procedure to test the data before releasing to the tables?. Kindly have a look

Comment 5

[Anthony Miyaguchi [:amiyaguchi]]

I've attached to the email thread comparing the daily search numbers for 20170615 against the topline-weekly search counts over the week of 20170611. They contain the same volume of searches. [1] 

I think you may be using the monthly dataset, which unfortunately slipped through the cracks. It currently isn't filtering against in-content searches. You can follow bug 1386639 for updates the monthly rollups. 


[1] https://metrics.services.mozilla.com/diagnostic-data-viewer/?dataset=topline-weekly

Comment 6

[Anthony Miyaguchi [:amiyaguchi]]

There are also a series of unit tests that we apply to the search_rollups (https://github.com/mozilla/python_mozetl/blob/master/tests/test_search_rollup.py). Testing data is a difficult procedure, because there is no such thing as ground truth. However, there are various datasets that have some historical weight to them, the topline_summary (topline-weekly, topline-monthly) being one of those.

There probably should be some sort of procedure to validate some simple statistics about the dataset (row numbers, summary statistics), but its very difficult to track explicit changes in the data.

I've thought of some ways to do data validation via historical data. This is in the roadmap for some other data sets like churn topline.

Comment 7

[arana]

Hi Anthony,

This sounds great. One suggestion I have is to implement simple rollups for data like total search volume in a month, total unique users in a month and see the % difference from last month. In case of a sudden rise/fall (>25%) an alert goes off which demands rechecking the data or a business reason for the change (I can provide that). Let me know what you think