973612 - Import page visit stats from Google Analytics and make it available to the API

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: API, defect)

Description

[Florian Scholz (Open Web Docs)]

Especially for translation status pages like [1] it would be really helpful to have visit counts to prioritize the l10n work based on visitors.

For each document a "visits" field could be added and a (monthly) script import from Google Analytics could populate this field with data. We don't really need live data here. A "visits" field could then also be added to the $json of a document, so that we can use KumaScript to work with this data.

Les, Luke? Is there a way with Google Analytics to get us this data like described above?

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Doc_status/ja

Comment 1

[Florian Scholz (Open Web Docs)]

Needinfo'ing Les and Luke.

Comment 2

[Jean-Yves Perrier [:teoli]]

Note that we don't need live values. Values updated monthly or even quarterly are perfectly fine.

Comment 3

[Luke Crouch [:groovecoder]]

Yes, there's a way and we should be able to re-use code like SUMO has for this:

https://github.com/mozilla/kitsune/blob/master/kitsune/sumo/googleanalytics.py

Comment 4

[Ricky Rosario [:rrosario, :r1cky]]

You will need to file an WebOps bug to get PyOpenSSL==0.13 installed on the servers. The fun part is that RHEL has 0.10 installed and WebOps will need to jump through some hoops for you. See all the gory details we went through in bug 830496

Comment 5

[Julien G]

After (quickly) digging into kitsune's code,  found the basic bricks for GA/metrics there:

1. getting GA data in [1] (see #3) : This uses OAuth2client [2] (see #4 from :rrosario on this). Fecthing is done on different periods : 7 days / 30 days / 90 days / since the beginning. Only the en-US views are counted (this may be enough based on the hypothesis that en-US pages are the most visited pages and "give" the trends) (see also [3] and [4])
2. storing this data in the database in the table 'dashboards_wikidocumentvisits' ([5])
3. querying/creating a view for the content (the query uses info like tags, removes non-relevant pages ) ([6] and [7])

This allows to 
- give a ratio dashboard ala "top 100/50/20" gauge to the final user for l10n
- give a freshness dashboard to English speaking contributors 
(see picture attachments after)

needinfo'ing Luke for the following:
- is GA data available in a self-made kuma environment (i.e. one that I could make out of github and so on)?
- are we collecting GA data about things like "someone went to the [locale] version of the page but was redirected to the en-US version" or simply views per locale for a page?

[1] https://github.com/mozilla/kitsune/blob/master/kitsune/sumo/googleanalytics.py
[2] https://pypi.python.org/pypi/oauth2client
[3] https://github.com/mozilla/kitsune/blob/master/kitsune/dashboards/cron.py
[4] https://github.com/mozilla/kitsune/blob/master/kitsune/dashboards/models.py
[5] https://github.com/mozilla/kitsune/blob/master/scripts/schema.sql (goto dashboards_wikidocumentvisits definition)
[6] https://github.com/mozilla/kitsune/blob/master/kitsune/dashboards/readouts.py
[7] https://github.com/mozilla/kitsune/blob/master/kitsune/dashboards/views.py

Comment 6

[Julien G]

Attachment: dashboard available to the L10N contributors

We can see the "Top 50/20..." on this screenshot. I am not saying this is something we need in doc status pages.

Comment 7

[Julien G]

Attachment: Dashboard available for English speaking contributors

Comment 8

[Julien G]

(In reply to Jean-Yves Perrier [:teoli] from comment #2)
> Note that we don't need live values. Values updated monthly or even
> quarterly are perfectly fine.

Do we want absolute values like 'nb of visits since forever' and/or also 'fresh visits' (like last 30 days) to react on "hot" pages? I don't know if this is relevant...

Comment 9

[Les Orchard [:lorchard]]

> needinfo'ing Luke for the following:

lorchard is not luke :)

Comment 10

[Julien G]

(In reply to Les Orchard [:lorchard] from comment #9)
> > needinfo'ing Luke for the following:
> 
> lorchard is not luke :)
mea culpa maxima :$ ^^

Comment 11

[Jean-Yves Perrier [:teoli]]

(In reply to Julien G from comment #8)
> (In reply to Jean-Yves Perrier [:teoli] from comment #2)
> > Note that we don't need live values. Values updated monthly or even
> > quarterly are perfectly fine.
> 
> Do we want absolute values like 'nb of visits since forever' and/or also
> 'fresh visits' (like last 30 days) to react on "hot" pages? I don't know if
> this is relevant...

Visits over the last month (either 1st - 1st or sliding month, whatever is the most easy for the devs) is enough here (last = last at the point of the latest update, as we don't need live values)

Comment 12

[Luke Crouch [:groovecoder]]

+1 :teoli

Comment 13

[BMO Automation]

MDN Web Docs' bug reporting has now moved to GitHub. From now on, please file content bugs at https://github.com/mdn/sprints/issues/ and platform bugs at https://github.com/mdn/kuma/issues/.