730707 - kumascript: Complete a good-enough MindTouch-compatible API for templates

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Editing, defect)

Description

[Les Orchard [:lorchard]]

Templates on MDN use the objects and methods exposed by DekiScript. The kumascript service also exposes APIs that can work largely the same way. Work through converted templates and ensure that enough of a compatibility layer is implemented so that as few manual changes need to be done as possible.

http://developer.mindtouch.com/en/docs/DekiScript/Reference

Comment 1

[Les Orchard [:lorchard]]

Thinking out loud, since I've mentioned this in passing a few times. We need to support things like this:

http://developer.mindtouch.com/en/docs/DekiScript/Reference/Wiki_Functions_and_Variables#Wiki_Variables

So, one of the things this feature will need is a way to pass document and request context from Kuma to Kumascript. Rather than cram this into the query string, I've got a notion to do this all in custom request headers. 

This includes things such as:

* current logged in user (if any) (Kumascript-Var-Username: lmorchard)
* document locale (Kumascript-Var-Locale: en-US) 
* document slug (Kumascript-Var-Slug: DOM/Storage)
* document title (Kumascript-Var-Title: DOM Storage)
* document translation parent & siblings (see bug 735888)

One challenge with this approach is that the safest encoding for HTTP headers is plain ASCII [1]. That's why the FireLogger protocol used in bug 730708 employs base64 in header-based error messages, so anything non-ASCII gets packaged safely. 

Of course, the plain ASCII restriction is mainly about browser interoperability (glaring at MSIE). Since we're talking about Kuma-to-Kumascript interoperability, we might be okay with stuffing utf8 strings in there.

[1] http://stackoverflow.com/questions/1361604/how-to-encode-utf8-filename-for-http-headers-python-django

Comment 2

[Les Orchard [:lorchard]]

I'm also tempted to just follow something like the FireLogger protocol for variables. That is: JSON-encode a big blob of variables, base64 encode it, split it up into lines, stuff the lines into headers.

Could do this for the whole set of variables, or one base64'd JSON blob per variable. The former might make for a nicer Vary: header (ie. Vary: X-FireLogger, Kumascript-Vars)

Comment 3

[Luke Crouch [:groovecoder]]

for sure users, slugs, and titles will all have utf8 characters.

Personally I'd vote for http://greenbytes.de/tech/webdav/rfc2231.html just because it's Mozilla, but I've never done either approach.

Comment 4

[Les Orchard [:lorchard]]

Hmm, yeah, I've got RFC 2231 open in another tab but haven't totally absorbed it yet. I think the only issue is limited browser support - but that's probably not all that troublesome here.

The interesting thing will be if the Requests lib on the Python side, and the requests module on the node side support it.

Comment 5

[Les Orchard [:lorchard]]

Mental note for this:

I've started sketching out the APIs in api.js in the kumascript repo. But, I wonder if it might be more interesting to let templates include other templates from the wiki as common libs, and move the api.js code into one of those.

Then, we could fill in missing API shims as we fix templates, without a need for a code push to the kumascript service.

Comment 6

[Luke Crouch [:groovecoder]]

Put the top 10 templates and their pages in a state to show Sheppy.

Comment 7

[Les Orchard [:lorchard]]

Still working through the templates, but the pages I'm fixing will appear here:
    http://mdn.lmorchard.com/en-US/docs/tag/testday420

And the associate templates here:
    http://mdn.lmorchard.com/en-US/docs/tag/testday420-templates

Comment 8

[Les Orchard [:lorchard]]

Also for handy reference, these are the MindTouch API shims I'm editing:
    http://mdn.lmorchard.com/en-US/docs/tag/dekiscript

Comment 9

[Les Orchard [:lorchard]]

As an additional note: Those dekiscript tagged should be where implementation of API shims go, but I can't lift them out of my sandbox until bug 747137 works

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/262ee179a3964a87e3c03064752ce6f0fdc770e9
bug 730707: progress checkpoint

* Switch back to out-of-box <% %> EJS escape sequences instead of {% %},
  since the ACE editor interferes with that less and the EJS docs make
  more sense that way.

  * TODO: Maybe need a schematic migration to convert existing templates
    in dev boxes at least:

      update wiki_document
      set html=replace(replace(html, '{%', '<%'), '%}', '%>')
      where slug like '%Template:%';

      update wiki_revision
      set content=replace(replace(content, '{%', '<%'), '%}', '%>')
      where slug like '%Template:%';

* Don't process views of Template:* pages through KumaScript

* Don't apply Bleach to Template:* source

* Reduce required length of page title to 2, so "en" and "CSS" can be
  edited.

* Whitelist a few more elements used in pages for Bleach.

* Add more elements with style attribute allowed by Bleach, whitelist a
  bunch of inline styles used in existing pages.

* Improvements to kumascript error display, broken out into an include.
  Includes edit / new links for Template documents behind scripts.

* Enable plain-text syntax highlighter brush

* KumaScript update

https://github.com/mozilla/kuma/commit/7f2f0d2e8cc8bdf3e1a45fb17ca46ae67ace2771
bug 730707: Bugfix for macro migration

* Bugfix for macro migration, where markup appears in the middle of a
  parameter. This happens in a lot of `note` and `warning` macros.

https://github.com/mozilla/kuma/commit/24f464d41cef0f2937c787c9673d8ccf29f6febe
bug 730707: Bugfix case-sensitivity in locale detection and migration

https://github.com/mozilla/kuma/commit/716dd4a95e27d274a8b55faae5feca0628bd186f
bug 730707: Rework section ID generation

* More closely match MindTouch for generating anchor IDs in pages, to
  help prevent breaking intra-wiki anchor links

* Honor an explicit autogenerated ID override where the `name` attribute
  is present. That should help break fewer page anchor links on existing
  pages.

https://github.com/mozilla/kuma/commit/d37b4c651b96333bdc5c48f44adb13acd5aa9960
bug 730707: Title is not unique on documents.

* Remove vestigal title uniqueness enforcement that caused DB errors

https://github.com/mozilla/kuma/commit/395efccbda202bdb2dc64c3e3f5310d125790582
bug 730707: Send doc metadata to kumascript

* Kumascript now accepts env vars as headers with base64/utf8/JSON
  encoded data structures

* Assemble some env vars from document metadata to send in the request
  to kumascript

https://github.com/mozilla/kuma/commit/04ef5ebe8547ccb042f077089f14e9900352e04f
bug 730707: Accept ?include parameter on doc view to filter out class="noinclude" blocks

https://github.com/mozilla/kuma/commit/53809c213f7bc903f6eec1b81c8b6dceb1e8c5d5
bug 730707: Increase allowed document length from 100k to 300k

https://github.com/mozilla/kuma/commit/d7f420a316f6fa686b1c87e56cdaf91f68e70957
Merge pull request #164 from lmorchard/bug-730707-kumascript

Bug 730707 kumascript