1095113 - Since moving to the new HIVE metrics system statistics by language for locales including a region-code are missing

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Statistics, defect, P3)

Description

[Onno Ekker [:nONoNonO UTC+1]]

As noted in bug 1089358 comment 21 language statistics for locales including a region code (e.g. en-US, en-GB, pt-BR, etc.) are missing from the statistics by language, starting from October 21.

Comment 1

[David Larlet [:davidbgk]]

Note that we made an attempt to merge the plurality of locales during the process:
https://github.com/mozilla/olympia/blob/bc44dc27935f357cfcadfe65b9c2557559fbada5/apps/stats/management/commands/update_counts_from_file.py#L175

Now all locales will be lowercased and - replaces with _ so pt-BR will become pt_br and so on. Is that bug still relevant knowing that?

Comment 2

[Onno Ekker [:nONoNonO UTC+1]]

Well, the data is missing frmo the graph and csv/json data, but that's perhaps that the page is now showing en-us instead of en-US, while in fact it should be showing en_us? You can see that the missing data on the language statistics of my add-on Mail redirect: https://addons.mozilla.org/en-US/thunderbird/addon/mailredirect/statistics/usage/languages/?last=30

Comment 3

[Onno Ekker [:nONoNonO UTC+1]]

Attachment: Missing statistics for locales with region-code.png

Comment 4

[David Larlet [:davidbgk]]

Oh I didn't see the graph. Mmmh, to me it's related to bug 1089358 and bug 1093699. Once both will be resolved and pushed to prod that one should be resolved too. But I keep it opened to verify that it's the case given that I'm the author of that locale merge, thanks for declaring it.

Comment 5

[David Larlet [:davidbgk]]

Fixed in https://github.com/mozilla/olympia/commit/2da300ee00748a000ed3b2d1dd76f92205058953

Comment 6

[Onno Ekker [:nONoNonO UTC+1]]

I've checked both my own stats but the data is still missing. Does there need to be some backfilling or isn't this yet pushed to production?

I tried clearing my localstorage because I thought that might have been the problem (bug 1089358 comment 20), but that didn't make a difference (even though I tried it in Web Console, Browser Console and Error Console). After that I also looked up the stats for an add-on I'd never seen before, but there also the data for locales including a region code was missing.

Unfortunately the latest available stats at the moment are from november 19, so it might be that this bug is fixed correctly and the only thing thaat still needs to be done is to fill in the gaps...

Comment 7

[David Larlet [:davidbgk]]

(In reply to Onno Ekker [:nONoNonO UTC+1] from comment #6)
> I've checked both my own stats but the data is still missing. Does there
> need to be some backfilling or isn't this yet pushed to production?

The backfilling will be completely done at the end of next week. There is a lot of data to compute.

Comment 8

[Onno Ekker [:nONoNonO UTC+1]]

I see new data now starting from november 20, so new statistics for locales including region-code works. I don't see any backfilling happening yet, even though I've cleared my localStorage, but maybe that data only appears after all backfilling is done and you have rerun index_stats --date=2014-10-21:2014-11-20 ?