1333868 - DSMO-RS is not up to date

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

As of 2017-01-25, the data in download_stats only goes up to 2017-01-13:
https://sql.telemetry.mozilla.org/queries/2326#4291

and download_stats_errors only goes to 2017-01-23:
https://sql.telemetry.mozilla.org/queries/2325#4289

It was in much worse shape on 2017-01-23 when I asked robotblake about it:
> <agashlin> The last day with data for download_stats is 2016-12-20,
>    for download_stats_errors it's that big spike on 2016-11-25
>    (I assume the approx 2k/day is more like the expected counts)
> <agashlin> download_stats_errors_YYYYMMDD tables keep showing up,
>    but empty, and download_stats_YYYYMMDD stopped 2017-01-13
>    (which does have data)

but download_stats is not yet caught up and download_stats_errors is not keeping up.

Comment 1

[Romain Testard [:RT]]

We'll rely on this for onboarding funnelcake testing starting soon, any idea when it could be looked at?

Comment 2

[Blake Imsland [:robotblake]]

I applied a temporary fix to get new days processing again and am working on another full backfill due to missing data (null fields) as mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1335704.

That backfill will fix the missing days.

Comment 3

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

Looking at the two queries from the description again, there seems to be disruption at least from 2017-02-07 (when download_stats_errors volume drops), and no data since 02-16 in download_stats_errors and since 02-17 in download_stats.

We're currently trying to use download_stats_errors as a workaround for funnelcakes until bug 1333533 comes to fruition (see bug 1333533 comment 13), but I just took a quick count of download_stats_errors rows with version like 'v7-%' and found that there is almost nothing for days besides 02-08 and 02-09: https://sql.telemetry.mozilla.org/queries/3239/source#table

Is funnelcake data being directed into another table already?

Comment 4

[Blake Imsland [:robotblake]]

This is working now.