855485 - [socorro-crashtats] provide public API [tracker]

Status: RESOLVED FIXED

(Socorro :: Webapp, task)

Description

[Lonnen :lonnen]

It would be nice for crashkill, academics, scripted tools, etc to be able to grab Socorro info from an API.  It should filter out PII.

We can achieve this by creating a view in socorro crashstats that provides access to the models. We could dogfood it by using it to replace our existing (limited) collection of one-off json endpoints. We'd get cache for free, and could extensively 

Something like:
`/app/:model_name?argument=value&argument=value`

Comment 1

[Peter Bengtsson [:peterbe]]

We currently have code like this::

  class CrashesPerAdu(SocorroMiddleware):

    # Fetch records for active daily users.
    def get(self, **kwargs):
        required_params = [
            'product',
            'versions',
        ]
        possible_params = [
            'from_date',
            'to_date',
            'date_range_type',
            'os',
            'report_type'
        ]

We could refactor that a bit the point where we can introspect all classes and automatically generate a nice looking API documentation. All we'd need to do (says the optimist in me) is something like this::


  class CrashesPerAdu(SocorroMiddleware):
    """Fetch records for active daily users."""

    required_params = [
        'product',
        'versions',
    ]
    possible_params = [
        'from_date',
        'to_date',
        'date_range_type',
        'os',
        'report_type'
    ]

    def get(self, **kwargs):
        required_params = self.required_params
        possible_params = self.possible_params

Obviously we wouldn't be able to do much with types or default values but it'd at least make it slightly less necessary for users to avoid http://socorro.readthedocs.org/en/latest/middleware.html

Comment 2

[[DEACTIVATED] Adrian Gaudebert]

Is that a better solution than exposing the middleware? 

I would say yes as the django app already handles authentication and this solution is a lot simpler and faster to implement.

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to Adrian Gaudebert [:adrian] from comment #2)
> Is that a better solution than exposing the middleware? 
> 
> I would say yes as the django app already handles authentication and this
> solution is a lot simpler and faster to implement.

Not to mention caching! 

However... it's not that simple. Every GET request via our models.py gets cached nicely in memcache so we can bombard that pretty hard. 
But, what if someone changes the URL just a little bit, they could cause us a lot of unnecessary traffic and load. E.g.:

 curl https://crashstats.mozilla.com/api/CrashesPerAdu?product=X&version=00001
 curl https://crashstats.mozilla.com/api/CrashesPerAdu?product=X&version=00002
 curl https://crashstats.mozilla.com/api/CrashesPerAdu?product=X&version=00003
 etc. 

But that brings us back to an advantage of doing this "via Django" in that we can slap https://github.com/jsocol/django-ratelimit on it.

Comment 4

[Lonnen :lonnen]

Docs are nice and maybe even necessary, but I'd don't want to block the quick win of the functional component on the rabbit hole that is a new docs automation infra. Lets file that and other additional features as separate bugs and move forward incrementally.

Comment 5

[Peter Bengtsson [:peterbe]]

Working on a prototype now.

Comment 6

[Peter Bengtsson [:peterbe]]

Prototype is coming along nicely!
https://github.com/peterbe/socorro-crashstats/compare/bug855485-public-rest-api

A couple more models to wrap and test.
I think generating documentation out of this should be plucked out as a separate bug. Perhaps with blocking.

Also, the rate limiting should also be a separate bug.

Comment 7

[Peter Bengtsson [:peterbe]]

Pull request is up: https://github.com/mozilla/socorro-crashstats/pull/321

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro-crashstats

https://github.com/mozilla/socorro-crashstats/commit/d7ad4e4e043a5fc0d3696fd61b1f45e9ab56ff9f
bug 855485 - public rest API, r=rhelmer

Comment 9

[Peter Bengtsson [:peterbe]]

it's in prod!