994565 - Admins can use HTTP interface to access Webmaker user model

Status: RESOLVED FIXED

(Webmaker Graveyard :: Engagement Ladder, defect)

Description

[Adam Lofting (:adamlofting)]

This is the blocker/required-plumbing for several of the "counting contributors" tasks so I'm moving it into a separate bug.

* HTTP interface to our user model so that adam's contributor dashboard can get data from it, according to this API: https://docs.google.com/a/mozillafoundation.org/document/d/16Sas-dbBzSftWqacYhFRojjXLCkAXu6h2XxbkfALG-Q/edit#heading=h.nfgmi620bnkc

Comment 1

[Adam Lofting (:adamlofting)]

I would like it if this API could report on:

1) All contributors who meet our definitions regarding data on the user model as a de-duped total (in the format specified for the API)

2) Contributors who meet each of our definitions regarding data on the user model per activity (in the format specified for the API)

So I could query the API for: 

* 2a) Bug 991357 - count of users who have submitted a teaching kit
* 2b) Bug 993592 - count of users who have created an event
* 1) A total count which would include all users who have (submitted a teaching kit OR created an event OR done any of the future things we add to these counts of contribution)

--- 

We can work with either solution 1 or 2, but having both will give us a better understanding of the data.

Comment 2

[Matt Thompson (@OpenMatt :OpenMatt)]

(nothing to add here other than that I love how well tagged and documented all your stuff is)

Comment 3

[Jon Buckley [:jbuck]]

http://park-warden-staging.herokuapp.com/
http://park-warden-production.herokuapp.com/

Chris is going to back-fill it with some existing data.

Comment 4

[Adam Lofting (:adamlofting)]

This is awesome, but I'll need to be able to query by date to graph this over time.

I think I've added the necessary code to accept a date parameter:
https://github.com/jbuck/park-warden/pull/1

(but I haven't tested this as I don't have the database setup here to run this against).

Thanks again,
A

Comment 5

[Chris DeCairos (:cade)]

Attachment: https://github.com/jbuck/park-warden/pull/1

Comment 6

[Adam Lofting (:adamlofting)]

I've just pushed up some amends from jbucks initial feedback, but I'm still thinking about the implications of the data model.

Comment 7

[Adam Lofting (:adamlofting)]

New contributors is simple, and the current logic works. I.e. if your first contribution was within the 7 days prior of the query date, you were a new contributor in that period.

Active contribution gets weird, because the latest contribution date can end up being in the future of a query date.

For example:
2011: First contribution
2012: ??
2013: Last contribution

It's not possible to know with only first and last contribution dates if they were active in 2012.

But we do need to be able to query this by date to work with our model of Active contributors over time.

Comment 8

[Adam Lofting (:adamlofting)]

As a possible solution, could we store an array of contribution dates on bucheron?

Comment 9

[Chris DeCairos (:cade)]

<cade> adamlofting: I may have a hacky solution. we key a set of contributions on "<userId>:contributions" in redis - alongside the profile object "<userId>" - so all contributions by date are tracked
<cade> that way we always know to lookup the set by appending ":contributions" to the userid and use it as a key

Comment 10

[Chris DeCairos (:cade)]

I've added sets to track user contributions. they're keyed on "<id>:contribution".

I patched park warden (on MY master branch) to ignore set keys and deployed it to heroku.

Adam: we need to get your patch ready for merging. I'm not sure if we want to do the set analysis login in it.

Comment 11

[Chris DeCairos (:cade)]

I've added some commits to adams work. 

https://github.com/cadecairos/park-warden/pull/1

The latest contribution will now be considered the contribution closest to the query date (without being after it)

Comment 12

[Adam Lofting (:adamlofting)]

Thanks Cade, your commits look good to me but I'm getting an application error when I view the app on Heroku.

Let's catch up when timezones overlap today. We're very close to getting this data into the dashboard now.

Comment 13

[Chris DeCairos (:cade)]

Jon and I will troubleshoot the performance problems today.

Comment 14

[Jon Buckley [:jbuck]]

Okay, we've fixed the performance problems and you should be getting data back in 300ms!

http://park-warden-production.herokuapp.com/

Because we're creating aggregated data from each contribution event, it'll be accurate for past dates as well. Compare the following:

http://park-warden-production.herokuapp.com/?date=2014-01-01
http://park-warden-production.herokuapp.com/?date=2014-02-01
http://park-warden-production.herokuapp.com/?date=2014-03-01
http://park-warden-production.herokuapp.com/?date=2014-04-01

Comment 15

[Adam Lofting (:adamlofting)]

Awesome. This is working great.

Are we planning to backfill the user model with events data from 2013? 

Our 12 month active contribution period means people who did things in 2013 are active contributors into 2014.

Comment 16

[Matt Thompson (@OpenMatt :OpenMatt)]

* JBuck can you demo this in Wed's Webmaker.org call? 

https://teach.etherpad.mozilla.org/crossteam-Q2

Comment 17

[Jon Buckley [:jbuck]]

If I'm able, yes. If I'm still figuratively dead then cade or adam could demo too