881867 - Adapt & document Addon SDK docs publishing process for MDN

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: General, defect, P1)

Description

[Les Orchard [:lorchard]]

The Addon SDK docs have a built process based on a set of scripts:
    https://github.com/mozilla/addon-sdk-sdocs/

They're currently published to AMO. But, per bug 881859, we want to switch them to MDN. I'm a bit fuzzy on the exact process followed for these scripts, so this bug exists to get us to adapt & document this process for MDN.

First stage would be to get this into a state where anyone in IT could respond to a bug filed to manually update the docs on MDN.

Second stage would be to have the process totally scripted, so we can kick off an update in reaction to github commit webhooks.

So... next step: Can we get a walkthrough of updating the SDK docs on AMO, and talk about how to switch that to MDN?

Comment 1

[Les Orchard [:lorchard]]

For context, pirating :wbamberg's explanation from an email thread:

The SDK repository (https://github.com/mozilla/addon-sdk.git) ships with a set of Python scripts that built a pile of HTML files from the source. As Jeff says, these things all live under https://github.com/mozilla/addon-sdk/tree/master/python-lib/cuddlefish/docs, and the top-level script is called "generate.py".

To builds docs for a release, the relevant function is "generate_static_docs()", which is invoked from the command line using the "cfx sdocs" command. "generate_static_docs()" takes two arguments: the root of the SDK clone, and the version number to insert into the generated docs. It generates a bundle of HTML files and zips them up into a file called "addon-sdk-docs.tgz". That's a single set of docs for a release of the SDK.

((There's some extra complexity in there: up to and including SDK 1.14, we supported local docs: so you could call "cfx docs" and the SDK would build you a set of docs locally, then run a web browser pointed at them. This uses most of the same code as cfx sdocs, but there are some performance optimisations for the local docs: first, sdocs rewrites the links to be relative, so the doc release is portable, and second, cfx docs has some logic to avoid rebuilding the docs every time if nothing has changed. In practice we'll be able to throw all this away in future as we are no longer supporting locally generated docs.))

At the moment, actually building everything for a release is horr-i-ble. This is because we maintain all the old releases for all the old versions on AMO's server, and give all the releases a banner indicating that they are obsolete and providing a link to the latest version, so that people following links to old releases can understand that. For example:

https://addons.mozilla.org/en-US/developers/docs/sdk/1.12/modules/sdk/page-mod.html

...has a banner pointing people at:

https://addons.mozilla.org/en-US/developers/docs/sdk/1.14/modules/sdk/page-mod.html

Because AMO won't do this server-side, it's all done in the client. So each time I have to rebuild the previous version, marking it as obsolete, and build the current version. What makes it extra-gross is that I keep track of files that have moved or been removed, so that the "go to the latest version" link for:

https://addons.mozilla.org/en-US/developers/docs/sdk/1.6/packages/addon-kit/page-mod.html

...actually goes to ...

https://addons.mozilla.org/en-US/developers/docs/sdk/1.14/modules/sdk/page-mod.html

...and not:

https://addons.mozilla.org/en-US/developers/docs/sdk/1.14/packages/addon-kit/page-mod.html

That means that I need to rebuild *everything* when I make a release. Then I push it all to https://github.com/mozilla/addon-sdk-sdocs.git, tag it, and raise an IT bug. Then Jeremy can just check that out and blat it into https://addons.mozilla.org/en-US/developers/docs/sdk (I believe "blat" is the correct technical term).

A-nyway. How I'd expect this to work is (a bit) different depending on whether the SDK's doc system normalises itself, so that the interface it presents is Sphinx- or Jekyll-compatible, or whether it doesn't.

If it doesn't, what I'd expect to happen would be something like this:

* the SDK doc code adds a function that can look at a GitHub post-receive hook and tell the caller whether (1) this commit changed the docs and (2) whether it represents a new release of the SDK, and if so, what the release identifier (=Firefox version) is

* on a GitHub post-receive hook, some code running on some server wakes up, checks out the SDK repo, and calls this new function. If the docs changed or it was a new release, it builds a new set of docs using "generate_static_docs()", and passing in the release identifier if there was one. It then extracts the built zip file under the relevant location according to the URL scheme (like devmo/addon-sdk/development/... if there wasn't a release identifier, or devmo/addon-sdk/$release/... if there was).

This is the process I tried to outline here: https://wiki.mozilla.org/GitHub_Integration#How_could_we_do_it.3F.

In this way "making a release" would be done by machines, not by me, and we could delete https://github.com/mozilla/addon-sdk-sdocs.git, and I'd be happy. We'd also have up-to-the-minute docs builds, so the SDK docs would have a lot of the same immediacy that the MDN wiki has.

To deal with obsolete notices, I'm hand-wavingly assuming some magic could be done on the server to deal with this (i.e. when serving a page, figure out if there's a version of the same page in a newer release subtree). To deal with pages being moved, I'm not really sure what the right answer is. I would be OK with just redirecting people to a useful 404, as it doesn't happen often, and having some specific effort to deal with situations when we do move a lot of files (like when we abandoned packages a few releases ago). I'm hoping you have a better idea of sane ways to do this stuff to be honest.

If the SDK doc system becomes compatible with some existing framework, then how it would work would depend a bit on how that existing framework expects things to happen. But I can't imagine it would be too different.

Is anyone still awake?

Will

Comment 2

[Luke Crouch [:groovecoder]]

Jeremy, are you the best person to help us set our next technical steps to migrate this process over to an MDN process? Warning: if so, I'm going to pull you into a meeting for it. ;)

Comment 3

[Les Orchard [:lorchard]]

Can anyone on this bug help us move it forward? 

In a week or so (after I'm back from PTO), I can probably start poking around with the existing scripts in github. But, we could really use some help from the folks deploying the Addon SDK docs right now.

Comment 4

[John Karahalis [:openjck]]

James feels this is too big for the Kanban board, and I agree. Can someone break this effort down into smaller next steps, and mark those next steps as dependencies? Making progress on this effort (and tracking that progress) should be easier once we have those in place.

Comment 5

[Jeff Griffiths (:canuckistani) (:⚡︎)]

FYI, Will is on PTO until the end of *next* week.

Comment 6

[John Karahalis [:openjck]]

Jeremy: Any thoughts on comment 2?

Comment 7

[Jeremy Orem [:oremj]]

If you are going to be rolling this in to MDN, you'll need to talk to the WebOps team. jakem is the point person there.

Comment 8

[Les Orchard [:lorchard]]

(In reply to Jeremy Orem [:oremj] from comment #7)
> If you are going to be rolling this in to MDN, you'll need to talk to the
> WebOps team. jakem is the point person there.

Okay, yeah, jakem will probably be helping us get this running.

But, the first thing I think we need is some help laying out what you guys do *right now* to publish & update these docs. I'm guessing that will be jakem's first question.

So... can anyone on this bug help with nailing that down? Do we need to wrangle everyone into a meeting?

Comment 9

[Will Bamberg [:wbamberg]]

SDK docs are now actually on MDN.