912064 - Convert docs to ReStructured Text as canonical form, and put up on ReadTheDocs

Status: RESOLVED FIXED

(Bugzilla :: Documentation, enhancement)

Description

[Gervase Markham [:gerv]]

We want to convert the Bugzilla docs from Docbook to ReStructured Text and connect them to readthedocs.org. Thanks to lhenry for the excellent suggestion.

The process is as follows:

1) Convert the trunk docs (only) in a one-off process (using some scripts I've written)

2) Patch them up so we have a TOC etc. (the conversion is not 100% perfect)

3) rewrite makedocs.pl to use Sphinx to build HTML, LaTeX -> PDF, and TXT (if the relevant converters are available)

4) Check this all in on trunk, using the directory docs/en/rst, and remove the XML versions and checked-in built versions

5) Point http://bugzilla.readthedocs.org at the trunk repo (RTD supports Bazaar, so we don't have to wait for any SCM shift)

6) (Optional) Add a post-commit hook to ping RTD to rebuild on checkin

7) Update the "HTML" and "PDF" links on http://www.bugzilla.org/docs/ to point to RTD; retain the "API" link; eliminate the "TXT" link

This bug covers points 1 to 4.

Gerv

Comment 1

[Gervase Markham [:gerv]]

/me installs the prerequisites needed to build the current docs:

0 upgraded, 37 newly installed, 0 to remove and 5 not upgraded.
Need to get 445 MB of archives.
After this operation, 711 MB of additional disk space will be used.
Do you want to continue [Y/n]?

<sigh>

Gerv

Comment 2

[Frédéric Buclin]

Is there some tutorial somewhere to learn the new syntax (title, subtitle, bold, italic, list, note, warning, etc...)?

Comment 3

[Gervase Markham [:gerv]]

Attachment: 912064.diff

OK, here it is.

What you get:

- A conversion of Bugzilla's DocBook documentation to ReStructured Text, 
  wrapped in a Sphinx build system.
- Same structure and layout as the previous version of the docs.
- Compiles without errors or warnings, and without the need for 700MB of 
  libraries
- HTML, PDF and Text output formats
- No obvious markup problems.
- Looks pretty nice in HTML form.
- API documentation generation continues to work fine.

Limitations:

There are various things which are not in scope for fixing in this bug, because they are problems ported from the old version.

- The usage of various markup styles is not consistent. We need an RST style guide.

- The documentation seriously overuses the "Note: " construct.

- The internal formatting of the RST files, as text documents, could be significantly improved.

- Some of the documentation is out of date or inaccurate.

Also, the new process has the following issues:

- The LaTeX -> PDF process emits a lot of warnings. I've not looked at why - it could be just LaTeX being picky. The generated PDF isn't too bad, but probably needs work. 

There is a TODO file included which lists a few things one might want to do in the near future.

I'd call this a good basis to work from. I'm happy to tackle some of the outstanding issues over the next few weeks, as work allows. But as bzr doesn't support local commits easily, I'd like to get a first "good enough" version checked in, and then iterate.

To compile it, all you need is Sphinx <http://sphinx-doc.org/>.

Gerv

Comment 4

[Gervase Markham [:gerv]]

LpSolit: to answer your question, I've been using http://docutils.sourceforge.net/docs/user/rst/quickref.html , but there are lots of other ones out there. And we are going to need a short-ish style guide, which is on my list of things to do.

It's a lot simpler than DocBook :-)

Gerv

Comment 5

[Dave Miller [:justdave]]

For the record, the prereqs to build under DocBook were huge, too (and a pain in the ass to install correctly, even with a package manager).  That's part of the reason we had so little uptake on documentation edits.

Comment 6

[Dave Miller [:justdave]]

(In reply to Gervase Markham [:gerv] from comment #1)
> /me installs the prerequisites needed to build the current docs:
> 
> 0 upgraded, 37 newly installed, 0 to remove and 5 not upgraded.

Out of curiosity, what are the prereqs?

Comment 7

[Gervase Markham [:gerv]]

I think you may have the wrong end of the stick; my 711MB figure was the current docs (i.e. DocBook). 

Here's the output from my apt-get install of "xmlto":

The following NEW packages will be installed
  dblatex lacheck latex-beamer latex-xcolor lmodern luatex pgf
  preview-latex-style prosper ps2eps tex-common texlive texlive-base
  texlive-bibtex-extra texlive-binaries texlive-common texlive-doc-base
  texlive-extra-utils texlive-font-utils texlive-fonts-recommended
  texlive-fonts-recommended-doc texlive-generic-recommended texlive-latex-base
  texlive-latex-base-doc texlive-latex-extra texlive-latex-extra-doc
  texlive-latex-recommended texlive-latex-recommended-doc texlive-luatex
  texlive-math-extra texlive-pictures texlive-pictures-doc texlive-pstricks
  texlive-pstricks-doc tipa xmlto xsltproc
0 upgraded, 37 newly installed, 0 to remove and 5 not upgraded.
Need to get 445 MB of archives.
After this operation, 711 MB of additional disk space will be used.

It could be that some of this is needed for the new docs' PDF generation, so I need to be careful not to be too sarky.

Gerv

Comment 8

[Gervase Markham [:gerv]]

This is probably the best intro to Sphinx-flavoured reST: http://sphinx-doc.org/latest/rest.html .

I have written a documentation style guide, but let's get this patch done first. Then, we can add the style guide (and debate it :-) and when we have consensus, make whatever changes are necessary to make the docs match it.

Gerv

Comment 9

[Mark Côté [:mcote]]

Is it necessary to have so many directives in the Makefile?  I can't imagine most of them being used.  I'd rather have just a few of the obvious ones (html, singlehtml, latexpdf) to have a less cluttered file, and add others only if we need them.

Btw I am converting API docs over as well, and, in BMO's case, adding in WebService docs from extensions.  I think I will do that as a separate document, though, since each WebService will have its own page as a result of pod2rst.  I'll still build them all at the same time.

Comment 10

[Gervase Markham [:gerv]]

(In reply to Mark Côté ( :mcote ) from comment #9)
> Is it necessary to have so many directives in the Makefile?  I can't imagine
> most of them being used.  I'd rather have just a few of the obvious ones
> (html, singlehtml, latexpdf) to have a less cluttered file, and add others
> only if we need them.

The Makefile is auto-generated by the "sphinx-quickstart" command, which is what both Sphinx and RTD recommend you use to start a new project. So it makes sense to me not to change it.

> Btw I am converting API docs over as well

Ah, interesting. Why? Is the plan to no longer maintain the documentation as POD? Or do you just want to be able to feed it into the readthedocs.org platform so that it all looks consistent?

Gerv

Comment 11

[Frédéric Buclin]

(In reply to Gervase Markham [:gerv] from comment #10)
> Ah, interesting. Why? Is the plan to no longer maintain the documentation as
> POD?

It makes sense to keep the POD to describe methods. This also lets 011pod.t detect if some methods are undocumented.

It has never been discussed to remove them.

Comment 12

[Mark Côté [:mcote]]

Yup mostly for consistency.  I also want to add in a bit more explanatory text, which is easiest by just adding a new section before all the automatically converted docs.

I would actually argue that it makes sense to change autogenerated things, which are meant to fulfill a wider set of expectations than any one project probably needs.  The only time I wouldn't change autogenerated stuff is if there's a system that automatically updates it, which I don't think is the case here.  It's like cutting & pasting an example file--might as well only keep the relevant things, for clarity and brevity.

For example, Django projects regularly cut out all the comments and unused things from the autogenerated settings.py files.

Comment 13

[Mark Côté [:mcote]]

And sorry to be explicit, we have no plans to remove the POD docs.  We discussed it briefly, but I think it makes more sense to just update the PODs to make them a bit clearer where necessary.

Comment 14

[Simon Green]

(In reply to Mark Côté ( :mcote ) from comment #13)
> And sorry to be explicit, we have no plans to remove the POD docs.  We

I'll +1 this. Red Hat modifies the POD (especially for the Web Services) to handle our customisations. I imagine BMO does the same too.

Comment 15

[Mark Côté [:mcote]]

One goal is actually to allow BMO to have its own documentation, particularly for extensions, so other installations will be able to benefit as well. :)

Comment 16

[Mark Côté [:mcote]]

Another comment: is there any real benefit to having the sphinx-build calls be in a Makefile, other than the fact that it was generated automatically?  I want to bundle the API docs as a separate document.  That entails either duplicating the Makefile or ripping stuff out of it into an include file.  Since the command line is actually pretty simple, I think we may as well just throw it all into makedocs.pl and do away with the Makefile.  What do you think of that idea?

Comment 17

[Dave Miller [:justdave]]

Comment on attachment 799060 [details] [diff] [review]
912064.diff

Patch doesn't apply cleanly...  did we really change that much in the docs in the last two weeks?

Reversed (or previously applied) patch detected!  Assume -R? [n] n
Apply anyway? [n] y
Hunk #1 FAILED at 1.
File docs/en/xml/administration.xml is not empty after patch, as expected
1 out of 1 hunk FAILED -- saving rejects to file docs/en/xml/administration.xml.rej
Reversed (or previously applied) patch detected!  Assume -R? [n] n
Apply anyway? [n] y
Hunk #1 FAILED at 1.
File docs/en/xml/installation.xml is not empty after patch, as expected
1 out of 1 hunk FAILED -- saving rejects to file docs/en/xml/installation.xml.rej
Reversed (or previously applied) patch detected!  Assume -R? [n] n
Apply anyway? [n] y
Hunk #1 FAILED at 1.
File docs/en/xml/modules.xml is not empty after patch, as expected
1 out of 1 hunk FAILED -- saving rejects to file docs/en/xml/modules.xml.rej
Reversed (or previously applied) patch detected!  Assume -R? [n] n
Apply anyway? [n] y
Hunk #1 FAILED at 1.
File docs/en/xml/using.xml is not empty after patch, as expected
1 out of 1 hunk FAILED -- saving rejects to file docs/en/xml/using.xml.rej

Comment 18

[Dave Miller [:justdave]]

bzr says we haven't touched those files since early August.  I'm confused now.

Comment 19

[Mark Côté [:mcote]]

Yeah the patch doesn't apply cleanly for me either.  However most of the errors don't matter.  All those failures in docs/en/xml/ are just when trying to delete files, but leaving them there doesn't hurt anything if you're just testing this out.  One error that does matter is that the content of docs/bugzilla.ent.tmpl is changed, but the file is not successfully renamed.  Just moving it to docs/definitions.rst.tmpl will allow you to run "make html" in docs/en/.  But yes, this patch needs to be fixed, hopefully taking into account my comments above, particularly comment #16. :)

Comment 20

[Gervase Markham [:gerv]]

Attachment: Patch v.2

Here's another attempt against trunk; hope it works better. It applies fine for me to a fresh trunk checkout using patch -p0.

mcote: I have no objection to your proposal to rip out the Makefiles, but can we do it as a separate step, please? I'd much rather not try, get it wrong, and spend ages in review due to that.

Gerv

Comment 21

[Mark Côté [:mcote]]

Yup, no problem.  I can prepare a separate patch for review after this one is committed.

Comment 22

[Frédéric Buclin]

Comment on attachment 814917 [details] [diff] [review]
Patch v.2

>=== renamed file 'docs/bugzilla.ent.tmpl' => 'docs/definitions.rst.tmpl'

>-<!ENTITY bz-ver "4.5">
>-<!ENTITY bz-date "2012-08-30">
>-<!ENTITY current-year "2012">

Why removing these constants from here? And why splitting information between this file and docs/en/rst/conf.py?



>=== added file 'docs/en/rst/about.rst'

>+This document is copyright (c) 2000-2012 by the various

Years shouldn't be hardcoded. That's why we have constants defined elsewhere.


>+This is the 4.5 version of The Bugzilla Guide. It is so named

Do not hardcode version numbers. It's impossible to track them every time we release a new version. Why not using |version|?


(I didn't look further. What just looking at how the patch looked like.)

Comment 23

[Gervase Markham [:gerv]]

(In reply to Frédéric Buclin from comment #22)
> >=== renamed file 'docs/bugzilla.ent.tmpl' => 'docs/definitions.rst.tmpl'
> 
> >-<!ENTITY bz-ver "4.5">
> >-<!ENTITY bz-date "2012-08-30">
> >-<!ENTITY current-year "2012">
> 
> Why removing these constants from here? And why splitting information
> between this file and docs/en/rst/conf.py?

It's been so long now that I can't remember... <polite cough>

> Do not hardcode version numbers. It's impossible to track them every time we
> release a new version. Why not using |version|?

Seriously, I'm happy to fix all this stuff; but this is a transformative change, so the idea was that we'd get a quick review on the principle and the rough form, and then tidy it up in further patches. I'm very happy to do that extra work, but it would be awesome to get this patch reviewed to give a basis from which people can work.

justdave?

Gerv

Comment 24

[Frédéric Buclin]

(In reply to Gervase Markham [:gerv] from comment #23)
> Seriously, I'm happy to fix all this stuff; but this is a transformative
> change

Does this mean you converted compiled documentation to RST instead of converting XML source files to RST? I thought the latter, in which case you could catch constants used there.

Comment 25

[Gervase Markham [:gerv]]

No, I converted the XML source, but the converter script I found does not do everything, so there was some manual fixup needed, and further work will need to be done.

Gerv

Comment 26

[Mark Côté [:mcote]]

Maybe we can change reviewer here if :justdave doesn't have the time?

Comment 27

[Gervase Markham [:gerv]]

:justdave told me on IRC just before Thanksgiving that he was reviewing it as we chatted, but I don't know how far he got... Dave?

Gerv

Comment 28

[Mark Côté [:mcote]]

FYI this patch has bitrotted again because docs/bugzilla.ent.tmpl has changed again.  However, you just need to delete it and create docs/definitions.rst with this content:

.. |min-perl-ver| replace:: 5.10.1
.. |landfillbase| replace:: http://landfill.bugzilla.org/bugzilla-tip/
.. |bzg-bugs|     replace:: http://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla component=Documentation

The patch marks bugzilla.ent.tmpl as having been renamed to definitions.rst despite the content being quite different.

Also, makedocs.pl should create the output directory for the API docs ($lang/build/html/api) before calling batch_convert(), so it fails unless you create it ahead of time.  Other than that it works fine if you have all the correct packages installed (are these documented somewhere? I figured them out by running it over and over).

Comment 29

[Mark Côté [:mcote]]

Er, meaning, in the attached patch makedocs.pl does *not* create that output directory, so it fails; thus the patch should be fixed before being committed.

Comment 30

[Gervase Markham [:gerv]]

Attachment: Patch v.3

One more time, with feeling. Asking for review from glob, as justdave is busy :-)

The new patch is against trunk, and has fixes for the issues in comment 28 and some of those in comment 22.

Gerv

Comment 31

[Dave Miller [:justdave]]

Comment on attachment 8361024 [details] [diff] [review]
Patch v.3

rmbp:docs dave$ perl makedocs.pl
Could not open definitions.rst.tmpl: No such file or directory at makedocs.pl line 62.

Comment 32

[Dave Miller [:justdave]]

I see a definitions.rst file, and thought this was just a typo in the open command or something, but when I went to change the filename being asked for, I notice there's a second call right under that one to open definitions.rst without the .tmpl suffix, so apparently it's really expecting both?

Comment 33

[Gervase Markham [:gerv]]

makedocs.pl opens definitions.rst.tmpl for reading, and definitions.rst for writing, and uses the .tmpl file to create the other one, with some additional stuff extracted from the code.

You need to run makedocs.pl from $BUGZILLA_HOME.

gerv@hare:/usr/src/bugzilla$ rm docs/definitions.rst
gerv@hare:/usr/src/bugzilla$ docs/makedocs.pl 
Creating API documentation...
...
<snip>
gerv@hare:/usr/src/bugzilla$ ls docs/
definitions.rst  definitions.rst.tmpl  en  lib  makedocs.pl  style.css  TODO  xsl
gerv@hare:/usr/src/bugzilla$ 

Gerv

Comment 34

[Dave Miller [:justdave]]

that doesn't seem to help any, same error.

There is no definitions.rst.tmpl file in the patch.  There is a definitions.rst though, which doesn't make sense if that file is generated.

Comment 35

[Gervase Markham [:gerv]]

Attachment: Patch v.4

Missed a file. justdave is actively reviewing as I write.

Gerv

Comment 36

[Gervase Markham [:gerv]]

Attachment: Patch v.5

Paper bag patch.

Gerv

Comment 37

[Dave Miller [:justdave]]

Comment on attachment 8361152 [details] [diff] [review]
Patch v.5

If you're using homebrew on Mac, the sphinx package in homebrew is *not* the one needed for the docs build, that one's a search engine. :)

homebrew does not have a python-sphinx package.  I used the following to install it:

sudo easy_install pip
sudo easy_install -U setuptools
sudo pip install sphinx

homebrew does not have a latex package.  They do have a stub for it which politely informs you that they can't build it because there is no 64-bit version, and building the 32-bit version won't use all of the existing packages as dependencies because they're all 64-bit.  They point you at http://www.tug.org/mactex/ to get MacTeX which is a binary distribution that contains the whole thing with dependencies embedded. The download package is 2.2 GB and "This install requires 4.33 GB of space" to install. You can save 411 MB by deselecting the GUI applications in the installer.

After installing MacTeX, I had to do this:

cd /usr/local/bin
sudo ln -s ../texlive/2013/bin/universal-darwin/pdflatex

to allow sphinx-build to find it.

My main complaints about this are where the output goes to.
1) HTML output ends up at docs/en/build/html instead of docs/en/html
2) Text output ends up at docs/en/build/text instead of docs/en/txt
3) PDF output ends up at docs/en/build/latex instead of docs/en/pdf
4) docs/en/build/latex ends up with a lot of leftover junk in the directory besides the PDF file. (looks like temp files from intermediate build steps that didn't get cleaned up).

It otherwise looks beautiful, and spot-checking several random places in the docs against the current production docs on the website shows everything lines up nicely.  The above stuff can all be fixed after checkin, let's get this in before it bitrots.

Comment 38

[Frédéric Buclin]

(In reply to Dave Miller [:justdave] ([e-mail]) from comment #37)
> My main complaints about this are where the output goes to.
> 1) HTML output ends up at docs/en/build/html instead of docs/en/html
> 2) Text output ends up at docs/en/build/text instead of docs/en/txt
> 3) PDF output ends up at docs/en/build/latex instead of docs/en/pdf

This is going to break all links pointing to the documentation, including bugzilla.org and all references to the documentation used within Bugzilla.

Comment 39

[Dave Miller [:justdave]]

I think the intention was to fix it to output to the correct places, but that was coming in a separate patch from mcote from what I understood.

Comment 40

[Gervase Markham [:gerv]]

(In reply to Frédéric Buclin from comment #38)
> This is going to break all links pointing to the documentation, including
> bugzilla.org and all references to the documentation used within Bugzilla.

...which is why I have an updated patch which eliminates the extra /build/ directory, which I will attach and check in tomorrow :-) 

mcote said he might do a patch to eliminate the extra level of indirection caused by the makefiles; he thinks we can run the necessary commands directly from the Perl buildscript, as we don't need most of what the Sphinx makefiles give us.

Gerv

Comment 41

[Gervase Markham [:gerv]]

Committing  to: bzr+ssh://bzr.mozilla.org/bugzilla/trunk/
added docs/en/rst/conf.py
deleted docs/en/images/callouts
deleted docs/en/images/callouts/1.gif
deleted docs/en/images/callouts/2.gif
deleted docs/en/images/callouts/3.gif
added docs/en/rst/security.rst
missing docs/en/README.docs
deleted docs/en/README.docs
added docs/TODO
modified docs/style.css
added docs/en/rst/modules.rst
deleted docs/en/images/tip.gif
deleted docs/en/xml
deleted docs/en/xml/Bugzilla-Guide.xml
deleted docs/en/xml/about.xml
deleted docs/en/xml/administration.xml
deleted docs/en/xml/conventions.xml
deleted docs/en/xml/customization.xml
deleted docs/en/xml/gfdl.xml
deleted docs/en/xml/glossary.xml
deleted docs/en/xml/index.xml
deleted docs/en/xml/installation.xml
deleted docs/en/xml/modules.xml
deleted docs/en/xml/patches.xml
deleted docs/en/xml/security.xml
deleted docs/en/xml/troubleshooting.xml
deleted docs/en/xml/using.xml
added docs/en/rst/glossary.rst
deleted docs/en/images/warning.gif
deleted docs/bugzilla.ent.tmpl
added docs/en/rst/using.rst
deleted docs/xsl
deleted docs/xsl/bugzilla-docs.xsl
deleted docs/xsl/chunks.xsl
deleted docs/xsl/nochunks.xsl
deleted docs/xsl/pdf.xsl
added docs/en/Makefile
added docs/en/rst/about.rst
deleted docs/en/images/caution.gif
added docs/definitions.rst.tmpl                                                                                                                                                                             
renamed docs/en/rel_notes.txt => docs/en/historical_rel_notes.txt
deleted docs/en/images/note.gif
added docs/en/rst/customization.rst
added docs/en/rst/installation.rst
added docs/en/rst/gfdl.rst
added docs/en/rst/patches.rst
added docs/en/rst/troubleshooting.rst
modified .bzrignore
added docs/en/rst/_static/stuff.css
added docs/en/make.bat
added docs/en/rst/index.rst
added docs/en/rst/administration.rst
modified docs/makedocs.pl
added docs/en/rst
added docs/en/rst/_static
Committed revision 8871.

Yay! \o/

Gerv

Comment 42

[Frédéric Buclin]

(In reply to Gervase Markham [:gerv] from comment #41)
> renamed docs/en/rel_notes.txt => docs/en/historical_rel_notes.txt

Not sure why this file has been renamed. We moved to HTML release notes in Bugzilla 3.0, so this text file was last used for Bugzilla 2.22. We could as well kill this file, IMO.

Comment 43

[Gervase Markham [:gerv]]

I renamed because I thought people would object to killing it. If you don't mind, I will :-)

Gerv

Comment 44

[Frédéric Buclin]

Comment on attachment 8361152 [details] [diff] [review]
Patch v.5

>=== modified file 'docs/makedocs.pl'

>-    $converter->add_css('./../../../style.css');
>+    $converter->add_css('./../../../../style.css');

No idea why this change has been made, but it's wrong, so I reverted it.

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
modified docs/makedocs.pl
Committed revision 8872.

Comment 45

[Frédéric Buclin]

The formatting of the glossary is broken for the 'apache' entry, both in the HTML file and in the PDF file. I fixed that:

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
modified docs/en/rst/glossary.rst
Committed revision 8873.

Comment 46

[Gervase Markham [:gerv]]

I removed historical_rel_notes.txt.

Gerv

Comment 47

[Mark Côté [:mcote]]

FYI my patch is mainly to generate API documentation in RST format as well, and in particular to include API docs from extensions.  I'll file a bug for that shortly.

Comment 48

[Frédéric Buclin]

Added to relnotes for 5.0rc1.