1221648 - Snippet Stats Are Not Being Reported As Of 10/29

Status: RESOLVED FIXED

(Snippets :: Service, defect)

Description

[Ben (:bensternthal)]

Attachment: Screen Shot 2015-11-04 at 9.09.48 AM.png

Jean reported that she is not seeing snippet impression data as of 10/29 (see screenshot).

I checked my about:home and snippets are still pinging the stats server, AFAI no changes there and it appears to be working properly:

> https://snippets-stats.mozilla.org/foo.html?snippet_name=5432&locale=en-US&country=&metric=impression&campaign=Fall2015

So my first guess is that something is up on the collection side.

Josephine can you check things on your end. Because of fall campaign this is a really important time for this data.

Comment 1

[[:josephine]]

Attachment: snippet live data - Oct 26 to Nov 3.csv

Checking with data team...
I can see data coming in up to Nov 3 but the numbers are significantly low. See attached for live data in vertica.
cc-ing Matt Pressman in the data team to check.

Comment 2

[[:josephine]]

> I can see data coming in up to Nov 3 but the numbers are significantly low.

To clarify - data is low from Oct 29 to Nov 3.

Comment 3

[Ben (:bensternthal)]

One additional piece of info.. the move from PHX1 to SCL3 happened on October 28th.

Comment 4

[Sheeri Cabral [:sheeri]]

There seems to be an issue with log copying. Preliminarily, it looks like we have the data we need, we just need to copy logs and reprocess. Will keep you up to date.

Comment 5

[Sheeri Cabral [:sheeri]]

Confirmed, there is 0 data loss, we have all the weblog files, they just weren't copied to the right place, due to needing a netflow. We are working on getting that netflow open, getting the logs copied, and processing logs.

Comment 6

[Sheeri Cabral [:sheeri]]

Sorry I did not update this yesterday. Current status - backlog is in hadoop, needs to be uploaded to Vertica:

15722807        2015-10-20
15597560        2015-10-21
16009964        2015-10-22
17387374        2015-10-23
12853824        2015-10-24
13005283        2015-10-25
18931827        2015-10-26
19676455        2015-10-27
16713051        2015-10-28
17976371        2015-10-29
16475625        2015-10-30
11746977        2015-10-31
 9730640        2015-11-01
14425448        2015-11-02
15758953        2015-11-03
23026493        2015-11-04
19949065        2015-11-05
Time taken: 90.921 seconds

Comment 7

[Sheeri Cabral [:sheeri]]

Vertica has been uploaded:
Fennec:
dbadmin=> select snippet_impression_date, count(*) from snippet_count_fennec where snippet_impression_date between '2015-10-20' and '2015-11-04' group by 1 order by 1;
 snippet_impression_date |  count  
-------------------------+---------
 2015-10-20              | 1717050
 2015-10-21              | 1682788
 2015-10-22              | 1606841
 2015-10-23              | 1504504
 2015-10-24              | 1044606
 2015-10-25              |  822631
 2015-10-26              |  813580
 2015-10-27              | 1456073
 2015-10-28              | 1165944
 2015-10-29              | 1147565
 2015-10-30              | 1172929
 2015-10-31              | 1127332
 2015-11-01              |  824546
 2015-11-02              |  192688
 2015-11-03              | 1191615
 2015-11-04              | 1200438
(16 rows)

Desktop:
dbadmin=> select date, count(*) from snippet_count where date>='2015-10-20' group by date order by date;
    date    |  count  
------------+---------
 2015-10-20 | 1360593
 2015-10-21 | 1315296
 2015-10-22 | 1302191
 2015-10-23 | 1259259
 2015-10-24 | 1163127
 2015-10-25 |  898556
 2015-10-26 |  868916
 2015-10-27 |  868934
 2015-10-28 |  864008
 2015-10-29 |  947113
 2015-10-30 | 1080139
 2015-10-31 | 1047688
 2015-11-01 |  997220
 2015-11-02 | 1087929
 2015-11-03 | 1119953
 2015-11-04 | 1239821
 2015-11-05 | 1514883
(17 rows)

Comment 8

[Sheeri Cabral [:sheeri]]

https://dataviz.mozilla.org/views/SnippetMetrics_0/TotalSnippetImpressions#6 is showing the drop-off disappeared.

Comment 9

[Jean Collings]

Thanks Sheeri!