910403 - Create endpoint for delivering metrics from MakeAPI (for services like GeckoBoard)

Status: RESOLVED INCOMPLETE

(Webmaker Graveyard :: MakeAPI, defect)

Description

[Mike Beltzner [:beltzner, not reading bugmail]]

The objective is to be able to have endpoints that can be called (perhaps with an API key if we're worried about security or privacy) in order to yield the following usage metrics:

 - total number of Makes
 - Total Makes created in the past 24 hours
 - Remixes created in past 24 hours
 - Total Makes created in the past 7 days
 - Remixes created in the past 7 hours

This will require:

 - the creation of the endpoint(s)
 - queries to be written in order to generate the desired metrics, and
 - a JSON format for the metrics

There may be other metrics, but these would represent a good start!

Comment 1

[Zakeria Hassan]

Hi can you assign this bug to me so I can work on it. 

Thanks,
Zak

Comment 2

[Jon Buckley [:jbuck]]

Hi Zak, thanks for offering to work on this bug!

We love having people build Webmaker with us, but this isn't a good starter bug to dip your toes in. It's a big feature and the looking glass is foggy when we look at how to implement this feature.

I don't know what sort of bugs you find interesting, but if you find one you want to do come chat with us on irc.mozilla.org in #webmaker. We're typically there from 9-5 EST on weekdays. Hope to chat with you soon!

Comment 3

[Matthew Schranz [:mjschranz]]

This is a mentored bug being picked up by one of Dave's students. At this point I don't know of it being something that's a pressing need so it's perfectly fine letting him work on it Jon.

Adding Dave/Cade to the CCs.

Comment 4

[Chris DeCairos (:cade)]

To those interested in hacking on this bug, I'm going to describe how I see this bug coming together.

note: I may use some terminology you're not familiar with, if that's the case, ping me on IRC or email me and I'll help you out.

We're going to want to add in a handful of protected routes:

/metrics/makes/all
/metrics/makes/day
/metrics/makes/week
/metrics/remixes/all
/metrics/remixes/day
/metrics/remixes/week

Initially, we can set up a page that displays the metrics to logged in administrators. use the admin console as a model. `/admin/metrics` sounds like a nice place to put it. In the future we could protect the metrics routes with Hawk and allow authenticated calls to it by admin keys.

Code will need to be written so that each route can generate the appropriate query to ElasticSearch, and then process the results into the data we need. You're going to want to familiarize yourselves with Query DSL http://www.elasticsearch.org/guide/reference/query-dsl/ (no easy task, don't try to learn it all, try to find out what will generate the data we need. HINT: look into Facets)

Also important is to define how responses will look in JSON. Consistency is important, and we might want to talk with someone who knows about how data should be formed when passed to various dashboards that we use.

If someone can figure out how to test these, I will be forever in your debt. Actually, scratch that. no merge if no tests are provided :) (which is kind of hypocritical of me, but w/e)

Comment 5

[Zakeria Hassan]

Hi,
I'm working on the implementation and I think I came up with something. Just to make sure I can mimic the same make request's posted to: "/api/make" can someone provide me with 1 example of how the JSON used will look in production:

On the Make API page it displays this:
'{ 
  "maker": "username", 
   make: { ... } 
}';

What data belongs in the make field?

Comment 6

[Matthew Schranz [:mjschranz]]

(In reply to Zakeria Hassan from comment #5)
> Hi,
> I'm working on the implementation and I think I came up with something. Just
> to make sure I can mimic the same make request's posted to: "/api/make" can
> someone provide me with 1 example of how the JSON used will look in
> production:
> 
> On the Make API page it displays this:
> '{ 
>   "maker": "username", 
>    make: { ... } 
> }';
> 
> What data belongs in the make field?

To get an idea you can look at the model for a Make (basically, table description) seen here https://github.com/mozilla/MakeAPI/blob/master/lib/models/make.js#L38-L121.

Comment 7

[Zakeria Hassan]

Hi,

All endpoints are functioning correct. If you would like to test this you can use the following endpoints:

  /metrics/makes/all
  /metrics/makes/day
  /metrics/makes/week
  /metrics/remixes/all
  /metrics/remixes/day
  /metrics/remixes/week

>Initially, we can set up a page that displays the metrics to logged in administrators. use the admin console as a model. `/admin/metrics` sounds like a nice place to put it. In the future we could protect the metrics routes with Hawk and allow authenticated calls to it by admin keys.

Note: I'm finished with the backend code. I would like to make a pull request and submit my code to Mozilla. 

I'd like to ask Cade a few questions about the UI as well has admin keys. Perhaps we could create new bug for the UI for the metrics & Hawk Authentication.

Comment 8

[Chris DeCairos (:cade)]

@Zak: I've left some comments in-line with your pull request on GitHub. Feel free to email me or find me on IRC if you have any questions.

Comment 9

[Zakeria Hassan]

@Cade: I've addressed all comments and I've pushed my latest code onto GitHub. Now I'm just waiting for my pull request to get approved.

Thanks,
Zak

Comment 10

[Chris DeCairos (:cade)]

Can you add the link to the pull request as an attachment to the bug?

Comment 11

[Zakeria Hassan]

Attachment: These are the changes I made which I would like to merge

This is the latest pull request and it highlights all the changes I've made. I've run the 'npm tests' and everything passed. Also I fixed every styling comment you mentioned in the past pull request & all code problems. Thank you again. Sorry for having go back and forth with reviewing this bug.

Comment 12

[David Humphrey (:humph)]

Attachment: https://github.com/mozilla/MakeAPI/pull/153

Just cleaning up so that we have the pull request link vs. compare branch.

We talked in class, and he's still going to be doing a rebase, once he gets an r+.

Comment 13

[Zakeria Hassan]

@cade: I am done all required fix's for this patch. Including a UI to demonstrate that it works. I used native javascript so it should work cross platform even on Firefox/Safari/IE. Anyway let me know how I did. I'm excited to get my first bug push to Mozilla.

Thanks,

Comment 14

[Chris DeCairos (:cade)]

Comment on attachment 815369 [details] [review]
https://github.com/mozilla/MakeAPI/pull/153

I left comments in the PR for Zak about a month ago - have not heard back from him since.

Comment 15

[Zakeria Hassan]

Thank you for the review. I'm sorry I wasn't quick to respond. I will fix it now.

Comment 16

[Zakeria Hassan]

Hi Cade,

Now after looking through the bug. I realized what happened. I commented on 2013-10-10 which was 2 months ago and I did not get a reply until a month ago. And I fixed every issue except the UI portion. I think I could quickly fix the UI and rebase. Before I go forward can you let me know if this is the only fix left because I got a few reviews for this bug specifically and there seems to be more to fix after I fix what is requested. Just for my understanding is the UI portion all that is left and why isn't it possible for us to use the data table because you did mention in a previous comment the following:

> use the admin console as a model. `/admin/metrics` sounds like a nice place to put it.

Questions:
1) Is the code that queries ElasticSearch correct?
2) Is there anything else left to resolve this bug?

Comment 17

[Zakeria Hassan]

I am ready and willing to fix this bug. I will fix the UI issue and I will update the pull request. Sorry for the delay. Thank you for your patience. Please disregard my previous comment.

Comment 18

[David Humphrey (:humph)]

Zak, what's the status of this bug?

Comment 19

[Zakeria Hassan]

I fixed the backend code for this bug long time ago. The issue that remains is how I display metrics in the UI. Cade requested that I not use the datatable that is used in the /admin page. Since this was a patch from 3 months ago code has changed and as a result merge conflicts are occuring for me. I going to close the pull request and try to resolve this merge conflict. Once I have everything working then I will submit a new one. 

Thank you,
Zak

Comment 20

[Zakeria Hassan]

Comment on attachment 815369 [details] [review]
https://github.com/mozilla/MakeAPI/pull/153

Okay code has been pushed all the code in with. This bug is complete.

Comment 21

[Chris DeCairos (:cade)]

I will take a look at this asap, thanks Zak!

Comment 22

[Zakeria Hassan]

Comment on attachment 815369 [details] [review]
https://github.com/mozilla/MakeAPI/pull/153

Sorry I can't edit this : 
New pull request:
https://github.com/mozilla/MakeAPI/pull/157

Comment 23

[Chris DeCairos (:cade)]

Zak: Create a new attachment, and mark the old attachment as obsolete by selecting the old attachment in the "obsoletes" section of the new attachment creation form.

Comment 24

[Zakeria Hassan]

Attachment: https://github.com/mozilla/MakeAPI/pull/157

https://github.com/mozilla/MakeAPI/pull/157

Comment 25

[Chris DeCairos (:cade)]

Comment on attachment 8349488 [details] [review]
https://github.com/mozilla/MakeAPI/pull/157

This got a review a while ago, I forgot to update this attachment. See github for review comments

Comment 26

[Zakeria Hassan]

Okay I'll work on this immediately. Thank you for the review.

Comment 27

[Chris DeCairos (:cade)]

The makeapi is no longer in operation, and isn't developed or maintained.

closing out this issue.