747158 - kumascript: Optimize wiki.pageExists

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Editing, defect)

Description

[Les Orchard [:lorchard]]

In building DekiScript API shims, I came up with an implementation that hits Kuma with a HEAD request for each call of wiki.pageExists to determine whether a page exists.

Naively, this makes sense. However, it turns out that some pages generate hundreds of wiki.pageExists calls, to decide whether to display a link to a page or a link to create a new page.

This could be made better by caching wiki.pageExists results from KumaScript in memcache. Then, it could be made better yet with a cron task on the Kuma side that warms up the cache straight from the DB.

In case the template gets lost, here's the current kumascript code:

    // Check if the given wiki page exists.
    // [See also](http://developer.mindtouch.com/en/docs/DekiScript/Reference/Wiki_Functions_and_Variables/Wiki.PageExists)
    // TODO: NEED CACHING HERE, BADLY!
    pageExists: function (path) {
        var p = kuma.url.parse(env.url, true);
        var base = p.protocol + '//' + p.host + '/en-US/docs/';           
        // ?raw is important, it makes Kuma refrain from hitting KumaScript
        // while being hit by KumaScript and avoids Recursive Madness
        var doc_url = base + path + '?raw';
        var opts = { method: 'HEAD', url: doc_url };
        var result = false;
 
        var f = new Future();
        request(opts, function (err, resp, body) {
            if (resp && 200 == resp.statusCode) { result = true; }
            f.return();
        });
        f.wait();
         
        return result;
    },

Comment 1

[Les Orchard [:lorchard]]

http://mdn.lmorchard.com/en-US/docs/en-US/Template:DekiScript:Wiki

Comment 2

[Les Orchard [:lorchard]]

Progress: https://github.com/mozilla/kumascript/pull/10

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/807ba319cd6e2ca15e44367bd32bcd1b4fd26662
Bug 747158 - kumascript: Optimize wiki.pageExists

* refresh_wiki_caches management command, which should be run from a
  crontab and will preload memcache with summary data useful for
  kumascript and possibly more in the future

* Fixes for encoding in kumascript response caching in Kuma

* Expose user agent Cache-Control header to templates, for future use in
  deciding whether or not to revalidate cached data

* kumascript JSON configs updated to use memcache

https://github.com/mozilla/kuma/commit/f42857aa64b455707b6d86f99e9bc4ec57efa239
Merge pull request #168 from lmorchard/bug-747158-optimize-pageexists

Bug 747158 optimize pageexists