873256 - MDN-flavored theme for reST docs in Sphinx / ReadTheDocs.org

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, defect, P1)

Description

[John Karahalis [:openjck]]

This bug describes one early step towards supporting outside sources of documentation.

(Quoting Les Orchard [:lorchard] from https://bugzilla.mozilla.org/show_bug.cgi?id=861981#c16)
> Build an MDN-flavored Sphinx theme and run something like the Marketplace
> API docs through it (because that's already written in reST). This will
> probably be a throwaway prototype at least in part, because we're also
> planning a redesign for MDN

Comment 1

[Les Orchard [:lorchard]]

I started some hacky work on this:

    https://github.com/lmorchard/mozilla-mdn-sphinx-theme

Here's a preview, while I wait my turn at getting a build through readthedocs.org:

    http://mdn-theme.static.lmorchard.com/

Comment 2

[Les Orchard [:lorchard]]

And, readthedocs.org has completed a build for me:

    https://django-valet-keys.readthedocs.org/en/latest/

Comment 3

[Les Orchard [:lorchard]]

This is also kind of neat:

    http://django-valet-keys.lmorchard.com/en/latest/

I added a CNAME to my DNS records to point a subdomain of lmorchard.com at readthedocs.org. 

This could just as easily by a subdomain of developer.mozilla.org, if that's how we wanted to play it.

Comment 4

[Luke Crouch [:groovecoder]]

Andy,

What do you think about changing the Firefox Marketplace API docs [1] to use this new MDN Sphinx theme [2]. Can you look at how django-valet-keys uses it [3] and how it renders [4] and give us feedback? Specifically ...

* What are your technical concerns? I don't do much sphinx, but setting the html theme and using a git submodule seems like a great approach to me.

* What are your UI/UX concerns? I worry that it's *too similar* to MDN - i.e., if we link from MDN to this kind of doc, a visitor loses some UI elements like the sign-in and edit buttons.

[1] http://firefox-marketplace-api.readthedocs.org/en/latest/
[2] https://github.com/lmorchard/mozilla-mdn-sphinx-theme
[3] https://github.com/lmorchard/django-valet-keys/tree/master/docs
[4] http://django-valet-keys.readthedocs.org/en/latest/

Comment 5

[Andy McKay]

+1 from me bug 889536

Comment 6

[David Walsh :davidwalsh]

What is the action we're looking to take within the scope of this ticket -- it is unclear.

Comment 7

[Les Orchard [:lorchard]]

(In reply to David Walsh :davidwalsh from comment #6)
> What is the action we're looking to take within the scope of this ticket --
> it is unclear.

Well, I made this by copying & pasting from view source while visiting MDN:

    https://github.com/lmorchard/mozilla-mdn-sphinx-theme

That's a Sphinx theme, but it's not great. It needs be made better. Part of the scope of this bug is working out all the things that are wrong with that theme and fixing them. I don't know how to better specify that without actually doing the work :)

You can see some examples using the theme here:

* http://django-valet-keys.readthedocs.org/
* http://django-teamwork.readthedocs.org/

So, the immediate next steps are probably to look at those examples, take notes on what needs improvement, iterate on the theme to make improvements. Of course, we also have a redesign coming up, so that will throw all this work in the air anyway.

Comment 8

[Les Orchard [:lorchard]]

In other words, the ultimate goal of this bug is to produce a Sphinx theme that can be applied to docs published on Read the Docs (or locally) that make them look like they're a part of MDN.

But, I don't think there's really a way to lay out a clear action plan without first iterating and experimenting. So, feel free to turn this into a tracking bug and file additional blocker bugs against it to break up the work as it's discovered.

Comment 9

[David Walsh :davidwalsh]

OK, that makes sense.  I've spoken to Luke about a few ideas and what I'm thinking is this:

Maintaining multiple files sets across these different sites would be a nightmare.  What I'd like to do is a multiple-part process:

1.  Markup the base template in the kuma REPO with HTML comments around structural elements.  For example:

<!-- header -->
( all the header html here )
<!-- /header -->


<!-- main -->
<section="mainContent">

</section>
<!-- /main -->

etc.

2.  The other sites/repos would periodically cURL to MDN and parse the desired pieces for the external site's template.  We could event add <!-- exclude --> comments to the main template to signal to the other sites to not include some HTML in their template, like the search box within the header.

Of course as I write that, I feel as though it may not be flexible enough.  The other option is adding "data-mdn-role" attributes to key elements within the base MDN template, and have each external site pick and choose which elements to use in its own base template. This allows the MDN dev team to avoid duplicating basic work on multiple repo's.

Would like thoughts from everyone before starting dev.

Comment 10

[Luke Crouch [:groovecoder]]

I much prefer semantic elements with data-* attributes over HTML comments. Ideally, we can use existing standards and specs as much as possible?

e.g., IIRC, one of the main things we want to keep consistent across the sites is the main developer navigation? Could we mark up the navigation lists on MDN with XOXO[1] to enable other sites/platforms to get it quickly and easily?

[1] http://microformats.org/wiki/xoxo

Comment 11

[Les Orchard [:lorchard]]

Not quite sure I get how this would work... Are you talking about building a scraper for each theme that visits developer.mozilla.org, pulls out delimited blocks, pastes them into the templates for each theme, and checks that into github? My initial hunch is that that might be more work than it's worth.

In my prototype theme, I'm referencing the JS / CSS / fonts / etc as external resources, so only the structural HTML itself needs to be checked into the theme repo. Is that HTML something we expect to change often?

Also, there will probably need to be tweaks between the main site and theme templates for Jekyll (bug 877221) / Sphinx (bug 873256) / Addons (bug 881864). We might be able to get away with a general base template for all three, and then put in the appropriate template magic for each system

Comment 12

[Les Orchard [:lorchard]]

Another wild idea that might be too meta to be easily understood:

Build some templates in Django that render as Jekyll / Sphinx / Addon SDK theme files. That way, we don't scrape anything, can use the common layout blocks & whatnot.

Comment 13

[Les Orchard [:lorchard]]

(In reply to Les Orchard [:lorchard] from comment #12)
> Another wild idea that might be too meta to be easily understood:
> 
> Build some templates in Django that render as Jekyll / Sphinx / Addon SDK
> theme files. That way, we don't scrape anything, can use the common layout
> blocks & whatnot.

Oh, and then the process to update a theme would be to run a shell script that

1) renders those templates with either cURL or a Django management command;
2) overwrite files in the repos;
3) commit & push to github.

Comment 14

[David Walsh :davidwalsh]

:groovecoder - There's nothing un-standard/semantic about using HTML comments for parsing.  Comments are easier as you can simply use String regex's to parse out content; with data-attributes you need to load the HTML as DOM and grab them that way.

:lorchard - You are correct in how you read my solution.  We will be changing structure a decent amount, or could be doing so, at least in the initial redesign months.  Although now that I think about, waffle flags could bust up the scraping idea.

I like the modified base template idea for external sites -- maintaing two templates within kuma should be easy enough.

Comment 15

[David Walsh :davidwalsh]

(In reply to Les Orchard [:lorchard] from comment #13)
> (In reply to Les Orchard [:lorchard] from comment #12)
> > Another wild idea that might be too meta to be easily understood:
> > 
> > Build some templates in Django that render as Jekyll / Sphinx / Addon SDK
> > theme files. That way, we don't scrape anything, can use the common layout
> > blocks & whatnot.
> 
> Oh, and then the process to update a theme would be to run a shell script
> that
> 
> 1) renders those templates with either cURL or a Django management command;
> 2) overwrite files in the repos;
> 3) commit & push to github.

Beautiful, love it.  Do we start duplicating the template now or wait on redesign?  I believe we're due to get something next week, although with the way things have gone thus far... (skeleton image)

Comment 16

[Luke Crouch [:groovecoder]]

Let's do it now. We can see just how easy it will be to maintain two templates in kuma when we have to make the changes for redesign.

Comment 17

[Les Orchard [:lorchard]]

(In reply to David Walsh :davidwalsh from comment #15)

> Beautiful, love it.  Do we start duplicating the template now or wait on
> redesign?  I believe we're due to get something next week, although with the
> way things have gone thus far... (skeleton image)

Could probably just start hacking on that now. For the Sphinx template, you could copy the wiki document template:

    https://github.com/mozilla/kuma/blob/master/apps/wiki/templates/wiki/document.html

Strip out all the logic, forms, etc and restructure / fill the content areas with Sphinx template tags like I used here:

    https://github.com/lmorchard/mozilla-mdn-sphinx-theme/blob/master/layout.html

A tricky part is that Sphinx template tags look just like Jinja template tags, so you'll need to escape [1] those char sequences like {{ '{{' }} and {{ '{%' }} to get them to output literally

[1]: http://jinja.pocoo.org/docs/templates/#escaping

Comment 18

[Les Orchard [:lorchard]]

Oh, and you'll probably also want to get set up with Sphinx and a project with docs using it. That way, you can run the build process and see what the theme looks like.

Off the top of my head, that should be as easy as:

sudo easy_install -U Sphinx
git clone https://github.com/lmorchard/django-teamwork.git
cd django-teamwork
git submodule update --init --recursive
cd docs
make html
open _build/html/index.html

Comment 19

[Luke Crouch [:groovecoder]]

Can I suggest we start with marketplace-api? :) It's one of the "external sources" we wanted to use as a pilot. (bug 889536)

Comment 20

[Les Orchard [:lorchard]]

Oh yeah, and my protip for hacking on the theme:

gem install kicker
kicker -c -e 'make html' .

And then, whenever you edit doc source or docs/_themes/mozilla-mdn-sphinx-theme/layout.html, the file change gets detected and kicks off the build process automatically.

Also, you can treat the docs/_theme/mozilla-mdn-sphinx-theme submodule as a full-on git checkout. So, you can fork it & add your fork as a remote. That, in turn, lets you push & submit pull requests & etc.

Comment 21

[Les Orchard [:lorchard]]

(In reply to Luke Crouch [:groovecoder] from comment #19)
> Can I suggest we start with marketplace-api? :) It's one of the "external
> sources" we wanted to use as a pilot. (bug 889536)

That's a good idea, but might be cumbersome as a first step. I mention my repos because they're small & already configured to use the theme submodule. 

The marketplace API docs are a part of zamboni itself. So, you'd have to...

1) Fork & clone zamboni
2) Add a submodule for mozilla-mdn-sphinx-theme under docs/_themes
3) Switch themes in docs/conf.py

Comment 22

[Les Orchard [:lorchard]]

Some status update on this bug. :davidwalsh got a first cut of this template-from-a-template process working:

    https://github.com/mozilla/kuma/commit/50d21ae684237da02e95a7b1110fa3b1507f92d5

I used that to update my prototype MDN/Sphinx theme:

    https://github.com/lmorchard/mozilla-mdn-sphinx-theme/commit/ff84a4bd7b7b788b507838f67b7a828e2155299b

And from there, I updated one of my projects and published the results to Read the Docs:

    https://django-teamwork.readthedocs.org/en/latest/

I think from here, we should try this theme out on a branch of the Marketplace API docs. Then, review the results and take notes on what needs improvement when applied to that more complex set. From those notes, maybe file blocker bugs against this one to implement improvements?

Comment 23

[Luke Crouch [:groovecoder]]

Sounds good to me. Let's close this and treat bug 889536 as our next action.