886612 - [email/backend] Set up automated documentation generation

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::E-Mail, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

We want automated documentation generation, ideally with some kind of automated mechanism that makes up-to-date documentation available on the web with minimal effort on our part.

Changing the doc syntax is a separate bug, bug 886610.

We need to look into the options, but our decision process weights heavily:
- Being implemented in JS itself.  AKA we don't want to have to download Java
- Understanding of documented complex types.
- Does not die if we start using ES6 features.

Less heavily, it would be nice if:
- The documentation tool supports some type inference and/or is able to perform other automated inference so that we don't really have to put annotations like @class on things that are pretty trivial for something that performs parsing.

Options discussed at our in-person meet-up include:
- Google closure compiler.  We don't like the Java aspect.  We like the type-flowing magic.  We are concerned about it not supporting newer JS features that we want to use, like generators.

- Yahoo's node-based tool.  We like that it uses node, not Java.  We like that it looks pretty.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Just had an impromptu IRC chat with :jrburke and :lightsofapollo after having becoming captivated by a dailyjs doctool link and done some additional poking around.

From discussion, it sounded like we don't really care about looking at pretty JavaDoc style structured docs and we'll mainly just look at the source code and make sure we write jsdoc tags.  However, a desire for higher-level implementation docs, preferably written in github-flavored-markdown with some block diagrams would be very nice.

I propose we do the following:

1) Use yuidoc (http://yui.github.io/yuidoc/) for automated documentation extracted from jsdoc-style syntax.  :lightsofapollo showed http://lightsofapollo.github.io/marionette_js_client/api-docs/classes/Marionette.Client.html as a demo and it looks pretty good to me and if we're already using it elsewhere, why not.

2) Use https://github.com/chjj/marked to transform markdown based docs into HTML.  Besides the suggested syntax highlighting plugin, I think it would also be handy to implement some simple diagram options.

https://github.com/bramp/js-sequence-diagrams makes pretty SVG sequence diagrams frmo a simple text format

Something for block diagrams would be good too.  http://blockdiag.com/en/blockdiag/examples.html#simple-diagram is neat, but is python and I don't want to get into virtualenv so much.  Maybe direct graphviz integration would be the simplest thing?


For build steps, :jrburke pointed out his https://github.com/volojs/volo-ghdeploy and :lightsofapollo indicated his demo uses a Makefile, see https://github.com/mozilla-b2g/marionette_js_client/blob/master/Makefile#L45

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

I have progress up in:
https://github.com/asutherland/gaia-email-libs-and-more/tree/autogen-docs

'make docs' runs yuidocs and generates some docs.  I converted a bunch of comments in mailslice.js and in some other files to see how well it worked.  Because yuidocs has a KISS viewpoint, I think some of the more complex types may require bogus .js files or the like.

The part that still needs to be finished to get these first steps landed is the conversion of markdown docs into HTML files with any additional magic.  For this bit, rather than writing a custom transformation step, it looks like https://github.com/maximebf/beautiful-docs does a good job of building an index and providing TOC-like functionality within a file as well as a way for hanging the syntax highlighting or diagramming off as a runtime pass.  Since that makes the 'view source' output look fine and none of those steps should be too expensive, that seems reasonable.  The only odd thing about beautiful docs is that it uses coffee-script, but I think that's a trivial thing for the time being.  

(:lightsofapollo indicated he's going to make a doc tool as a hobby project, so we can switch to that and its pure JS when the time comes :)

For diagramming, I changed my opinion frmo using js-sequence-diagrams to using jumly (https://github.com/tmtk75/jumly) with the demos at https://jumly.herokuapp.com/

jumly also uses coffee-script and uses that to directly implement a DSL of sorts.  While I'm not super-crazy about that, it does make sense to me and it does enable some styling and what not that seems useful.  The main reason jumly seems nice is that it makes pretty sequence diagrams with nested call stacks AND it also has robustness diagrams in-progress that seem good enough to serve as a means of doing block-diagrammy stuff.

My branch has bfdocs running okay, but I haven't integrated jumly into the theme.  I was trying to write-up an example .md file or two before I ran out of time on the weekend.

Comment 3

[James Lal [:lightsofapollo]]

VOTE Time!

Do we use the extended param syntax for objects:

@param myobject.property {Object}

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Useful data:

Here's closure compiler's super fancy syntax:
https://developers.google.com/closure/compiler/docs/js-for-compiler#types

Which jsdoc3 also supports:
http://usejsdoc.org/tags-type.html

yuidoc supports:
http://yui.github.io/yuidoc/syntax/index.html#param

and here's an example of an ad-hoc markdown snippet :lightsofapollo wrote:
* Creates a smtp and imap server pair.
*
* Options:
* - Object credentials: required in options.
* - Object credentials.username: username for server.
* - Object credentials.password: password for server.
* - Object options: options for imap server:
* - Array options.imapExtensions: extensions for server ['RFC2195'].
*
* @param {Object} options for server.
* @param {Function} callback [Error err, Object result].

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

My nominal vote is yes to: "@param myobject.property {type}"

I particularly vote for "@param name {type}" over "@param {type} name" because my brain gets a lot more context from the name and the type strings can potentially get a bit long.  I'd argue this is also more consistent with the traditional doxygen/javadoc syntax, although in that case the type is usually known statically from parsing.

I would also propose that for whitespace we do two-space indenting beneath the @param just so we don't lose too much space.

For example, on the streaming send branch, here's a pretty big mess of @param's in a row:
https://github.com/asutherland/gaia-email-libs-and-more/blob/ed43679f87566d05acda2854f1a90c3afd647041/data/lib/mailapi/mailslice.js#L4223

I am open to prettier options.  (Note that in the simple enum case in there with the strings, I'm not entirely happy with how I wrote it up, but it's a complex example to look at.)


If there's sufficient interest in going into something more hardcore like the jsdoc3 stuff, I have to admit I'm not opposed.


One other piece of info is that yuidoc doesn't actually support expressing things like "Array of Numbers".  See https://github.com/yui/yuidoc/issues/145

Comment 6

[BMO Automation]

Firefox OS is not being worked on