938915 - Create a one-off dump function for recommendation data

Status: RESOLVED FIXED

(Marketplace Graveyard :: API, defect, P1)

Description

[Wil Clouser [:clouserw]]

(This bug is actually close to bug 938887 so maybe read that one over too?)

Linas, from Telefonica Labs, is looking for some data so he can start crunching some numbers to give us rockin' recommendations.  From an email he is looking for a JSON output with rows of:

> one-way hashed user id (due to our strict privacy policy)
> id of installed app
> slug of installed app
> timestamp of install
> user's region
> user's language

A daily .tar.gz behind oauth would be ideal (and could probably be mostly copied from that other bug).  If the distribution is time consuming we could figure something else out since this is a one off, but this dump shouldn't be publicly accessible.

Comment 1

[Chuck Harmston [:chuck]]

Should this factor in "heart" data from bug 937278 when it lands?

Comment 2

[Wil Clouser [:clouserw]]

I'd like to get this out the door, so I'll say no, but we could certainly add more to it in the future.  Especially if we leave the code in to make dumps easy.

Comment 3

[Rob Hudson [:robhudson]]

We will use this JSON format for the user dumps:

  {
    "user": "<md5 hash>",
    "region": "br",
    "lang": "pt-BR",
    "installed_apps": [
        {"id": <app id>,
        "slug": <app slug>,
        "installed": "<installed date/time ISO 8601 format in UTC>"
      },
        {"id": 123,
        "slug": "twitter",
        "installed": "2013-11-18T08:50:24"
      },
      ...
    ]
  }

Comment 4

[Rob Hudson [:robhudson]]

Github pull request ready for review:
https://github.com/mozilla/zamboni/pull/1403

Comment 5

[Rob Hudson [:robhudson]]

https://github.com/mozilla/zamboni/commit/f7a4745

Comment 6

[Wil Clouser [:clouserw]]

How does this work?

Comment 7

[Rob Hudson [:robhudson]]

./manage.py cron dump_user_installs_cron

I was going to check on it tomorrow to see how it ran but we could run manually. The location of the dumps should be:
DUMPED_USERS_PATH = NETAPP_STORAGE + '/dumped-users'

Comment 8

[Wil Clouser [:clouserw]]

How do people get the dump?  Is there oauth?  URLs?

Comment 9

[Rob Hudson [:robhudson]]

(In reply to Wil Clouser [:clouserw] from comment #8)
> How do people get the dump?  Is there oauth?  URLs?

Nothing yet. What's the best approach? I thought we might not want URLs since these are somewhat sensitive. There is a VPN machine that was set up which could grab these via SSH/SCP? Or we can push them somewhere?

Comment 10

[Wil Clouser [:clouserw]]

looks like this didn't make it to prod so we'll have to wait another week for real data.  I wouldn't mind giving them a dump of -dev though so they could start parsing.  I filed bug 940722 to follow up.