780038 - Document edit API for MDN

Status: VERIFIED FIXED

(developer.mozilla.org Graveyard :: API, defect, P2)

Description

[Jeff Hammel]

User Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/17.0 Firefox/17.0
Build ID: 20120802030533

Steps to reproduce:

MDN should have a RESTful interface, as it did before transfer to kuma.

See http://developer.mindtouch.com/en/ref/MindTouch_API/POST%3Apages%2F%2F%7Bpageid%7D%2F%2Fcontents

In general, pages, or parts of pages may be taken from source documents and be mirrored to MDN.  As a for instance used for two projects, http://k0s.org/mozilla/hg/DocumentIt/file/ce7cbf3e08e1/document_it.py . This takes the markdown readmes from the source using a manifest and posts them to the appropriate pages on MDN.  (In some sort of ideal world, changes to MDN would be noted and conditionally taken upstream). 

Obviously just being able to post a document is a very small part of the mindtouch API.  In general the API is good. 

Auth for page changes should also be easy and not require e.g. an API key.  MDN credentials should be enough.  Ideally, a (e.g.) python package could be written to talk to the RESTful API.

Comment 1

[Les Orchard [:lorchard]]

We're talking about API access to MDN in toronto right this very moment. And some documentation is sorely needed for existing resources. But, some notes:

(In reply to Jeff Hammel [:jhammel] from comment #0)

> MDN should have a RESTful interface, as it did before transfer to kuma.

FWIW, we're very unlikely to ever have a clone of the MindTouch API.

What we do have so far in Kuma are an under-documented set of query parameters on docs to access content via non-humans, and a set of RSS/Atom/JSON feeds with machine-ready data.

No write access, yet, though. (Which I think is the main issue for this bug, thus the retitling.)

> In general, pages, or parts of pages may be taken from source documents and
> be mirrored to MDN.  As a for instance used for two projects,
> http://k0s.org/mozilla/hg/DocumentIt/file/ce7cbf3e08e1/document_it.py . This
> takes the markdown readmes from the source using a manifest and posts them
> to the appropriate pages on MDN.  (In some sort of ideal world, changes to
> MDN would be noted and conditionally taken upstream). 
>
> Obviously just being able to post a document is a very small part of the
> mindtouch API.  In general the API is good. 

So, what I think we're going to offer is simple: Document URLs will accept HTTP POST with content in the request body and create a new revision from that.

For example, POST to https://developer.mozilla.org/en-US/docs/Mozrunner will replace the contents. However, we also have a ?section parameter to constrain GET to content from a specific section - so POST with ?section can probably be made to replace content in only that section of the document.

> Auth for page changes should also be easy and not require e.g. an API key. 
> MDN credentials should be enough.  Ideally, a (e.g.) python package could be
> written to talk to the RESTful API.

I think we *will* be using API keys, at least for writes. The first problem is that auth for humans on MDN is now through BrowserID / Persona - there's no password any more. And the second problem is we'd like to be able to track API usage per application - and any give user could possibly maintain many applications.

So, my proposed solution so far is HTTP Basic Auth over SSL. Username will be user's email address, and password will be an API key managed in a personal settings panel. In the settings panel, you can create a simple record describing the app, and the site will generate a unique password for that app. Ideally, we'd also be able to present some metrics - eg. number of requests by the app in the past 24 hrs, etc.

Comment 2

[Jeff Hammel]

(In reply to Les Orchard [:lorchard] from comment #1)
> We're talking about API access to MDN in toronto right this very moment. And
> some documentation is sorely needed for existing resources. But, some notes:
> 
> (In reply to Jeff Hammel [:jhammel] from comment #0)
> 
> > MDN should have a RESTful interface, as it did before transfer to kuma.
> 
> FWIW, we're very unlikely to ever have a clone of the MindTouch API.

Sure.  I don't want or need a clone, just a way to post documents, or in the distant future possibly more.

> What we do have so far in Kuma are an under-documented set of query
> parameters on docs to access content via non-humans, and a set of
> RSS/Atom/JSON feeds with machine-ready data.
> 
> No write access, yet, though. (Which I think is the main issue for this bug,
> thus the retitling.)

Yep!

> > In general, pages, or parts of pages may be taken from source documents and
> > be mirrored to MDN.  As a for instance used for two projects,
> > http://k0s.org/mozilla/hg/DocumentIt/file/ce7cbf3e08e1/document_it.py . This
> > takes the markdown readmes from the source using a manifest and posts them
> > to the appropriate pages on MDN.  (In some sort of ideal world, changes to
> > MDN would be noted and conditionally taken upstream). 
> >
> > Obviously just being able to post a document is a very small part of the
> > mindtouch API.  In general the API is good. 
> 
> So, what I think we're going to offer is simple: Document URLs will accept
> HTTP POST with content in the request body and create a new revision from
> that.
> 
> For example, POST to https://developer.mozilla.org/en-US/docs/Mozrunner will
> replace the contents. However, we also have a ?section parameter to
> constrain GET to content from a specific section - so POST with ?section can
> probably be made to replace content in only that section of the document.

That sounds good to me.

> > Auth for page changes should also be easy and not require e.g. an API key. 
> > MDN credentials should be enough.  Ideally, a (e.g.) python package could be
> > written to talk to the RESTful API.
> 
> I think we *will* be using API keys, at least for writes. The first problem
> is that auth for humans on MDN is now through BrowserID / Persona - there's
> no password any more. And the second problem is we'd like to be able to
> track API usage per application - and any give user could possibly maintain
> many applications.
> 
> So, my proposed solution so far is HTTP Basic Auth over SSL. Username will
> be user's email address, and password will be an API key managed in a
> personal settings panel. In the settings panel, you can create a simple
> record describing the app, and the site will generate a unique password for
> that app. Ideally, we'd also be able to present some metrics - eg. number of
> requests by the app in the past 24 hrs, etc.

I believe that works for me.  Once this is implemented I am happy to test and/or write a general python client for this

Comment 3

[Les Orchard [:lorchard]]

Thinking this starts to get really important soon, to enable parties like devhub to inject updated docs into MDN. I'd also *just* started working on a developer API key management app, back at the end of the toronto trip. So, maybe I have a head start

Comment 4

[Les Orchard [:lorchard]]

I've started doing some work toward this:

https://github.com/lmorchard/kuma/tree/edit-api-780038

Comment 5

[Les Orchard [:lorchard]]

This is now in an initial pull request:

https://github.com/mozilla/kuma/pull/658

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/0125439d4977f8a7266e9f208bd5a5af87bef330
bug 780038: First draft of authkeys app

https://github.com/mozilla/kuma/commit/58b1e54275172f1803f243188fb8abba67757bb2
bug 780038: Wiki PUT API for edits, using authkeys

* New `authkeys` app to manage per-user API authentication keys

* Document view URLS accept PUT requests to edit content

* `revision.get_absolute_url()`, to build revision links

* `document.revise()`, makes a new Revision with selective changes

* `document.calculate_etag()`, builds a hash for full or section content

https://github.com/mozilla/kuma/commit/c8548fa0da8240091523c0a81c53e2994ae8bc4c
fix bug 780038 - rebase and refine

Comment 7

[raymond [:retornam] (inactive)]

verified fixed https://developer.allizom.org/en-US/keys/new

Comment 8

[Les Orchard [:lorchard]]

Started drafting some docs here:

https://developer.mozilla.org/en-US/docs/User:lmorchard/PUT-API

Plan to eventually move this to the Project:* namespace, once I've finished mangling a first draft.