423496 - (xmtlo) Move Documentation generation to xmlto from jade etc.

Status: RESOLVED FIXED

(Bugzilla :: Documentation, defect)

Description

[Colin Ogilvie [:cso]]

Move the documentation build process away from jade towards xmlto. 

This work will hopefully initially be completed on a branch of the documentation so that it can be monitored for a bit.

Comment 1

[Colin Ogilvie [:cso]]

I'm going to do this differently... the docs are on a branch (XMLTO_Documentation_Branch) and will be submitted as a patch once the branch builds.

Checking in makedocs.pl;
/cvsroot/mozilla/webtools/bugzilla/docs/makedocs.pl,v  <--  makedocs.pl
new revision: 1.20.4.1; previous revision: 1.20
done
Checking in style.css;
/cvsroot/mozilla/webtools/bugzilla/docs/style.css,v  <--  style.css
new revision: 1.1.4.1; previous revision: 1.1
done
Checking in en/xml/about.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/about.xml,v  <--  about.xml
new revision: 1.26.4.1; previous revision: 1.26
done
Checking in en/xml/administration.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/administration.xml,v  <--  administration.xml
new revision: 1.90.4.1; previous revision: 1.90
done
Checking in en/xml/Bugzilla-Guide.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/Bugzilla-Guide.xml,v  <--  Bugzilla-Guide.xml
new revision: 1.79.2.1; previous revision: 1.79
done
Checking in en/xml/conventions.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/conventions.xml,v  <--  conventions.xml
new revision: 1.12.4.1; previous revision: 1.12
done
Checking in en/xml/customization.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/customization.xml,v  <--  customization.xml
new revision: 1.44.4.1; previous revision: 1.44
done
Checking in en/xml/gfdl.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/gfdl.xml,v  <--  gfdl.xml
new revision: 1.11.4.1; previous revision: 1.11
done
Checking in en/xml/glossary.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/glossary.xml,v  <--  glossary.xml
new revision: 1.25.4.1; previous revision: 1.25
done
Checking in en/xml/installation.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/installation.xml,v  <--  installation.xml
new revision: 1.157.4.1; previous revision: 1.157
done
Checking in en/xml/integration.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/integration.xml,v  <--  integration.xml
new revision: 1.14.4.1; previous revision: 1.14
done
Checking in en/xml/modules.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/modules.xml,v  <--  modules.xml
new revision: 1.13.4.1; previous revision: 1.13
done
Checking in en/xml/patches.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/patches.xml,v  <--  patches.xml
new revision: 1.25.4.1; previous revision: 1.25
done
Checking in en/xml/security.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/security.xml,v  <--  security.xml
new revision: 1.19.2.1; previous revision: 1.19
done
Checking in en/xml/troubleshooting.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/troubleshooting.xml,v  <--  troubleshooting.xml
new revision: 1.13.4.1; previous revision: 1.13
done
Checking in en/xml/using.xml;
/cvsroot/mozilla/webtools/bugzilla/docs/en/xml/using.xml,v  <--  using.xml
new revision: 1.79.4.1; previous revision: 1.79
done

cvs commit: Examining xsl
RCS file: /cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/bugzilla-docs.xsl,v
done
Checking in xsl/bugzilla-docs.xsl;
/cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/bugzilla-docs.xsl,v  <--  bugzilla-docs.xsl
new revision: 1.1.2.1; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/chunks.xsl,v
done
Checking in xsl/chunks.xsl;
/cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/chunks.xsl,v  <--  chunks.xsl
new revision: 1.1.2.1; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/nochunks.xsl,v
done
Checking in xsl/nochunks.xsl;
/cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/nochunks.xsl,v  <--  nochunks.xsl
new revision: 1.1.2.1; previous revision: 1.1
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/pdf.xsl,v
done
Checking in xsl/pdf.xsl;
/cvsroot/mozilla/webtools/bugzilla/docs/xsl/Attic/pdf.xsl,v  <--  pdf.xsl
new revision: 1.1.2.1; previous revision: 1.1
done

Comment 2

[Colin Ogilvie [:cso]]

http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=XMLTO_Documentation_Branch&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-07-13+09%3A44%3A00&maxdate=2008-12-31+23%3A59%3A59&cvsroot=%2Fcvsroot

Comment 3

[Frédéric Buclin]

Attachment: patch, v1

Here is a fully functional patch. It correctly generates all desired formats: html, txt and pdf. The generated PDF file is half the size of the currently generated one and looks much nicer. We also have finer control on how generated files look like thanks to XSL stylesheets.

Tested on Mageia 1 and Fedora 16 beta.

Comment 4

[:glob ✱]

Comment on attachment 566594 [details] [diff] [review]
patch, v1

the documentation in en/README.doc needs to be updated to include the "xmlto" and "dblatex" pacakges, as these are new dependencies (and were not installed by default on my fedora 15 system, which is capable of building the old docs).

notes about the much improved pdf:

the pdf now has colour, which i think we should avoid (eg the page numbers on the contents are red, links are purple).  making everything black on white on the pdf will make it more printer friendly.

the sample for code on page 3 is corrupt:
  %%%%Thisisfile‘utf8x.def’,%%generatedwiththedocstrip
  </para>
it should look like:
  <para> Beginning and end of paragraph </para>

text in lists (eg. sections 5.3 and 5.5.4) isn't wrapped, and extends page the right margin.

html version:

the link colour is almost identical to the body text colour, making links hard to see for people who browser without "underline links" enabled.

text version:

code samples are no longer indented, which looks weird:

>   If you have installed the necessary Perl modules you can start
>   collecting statistics for the nifty Bugzilla graphs.
>bash# crontab -e

while examples have too much indentation, resulting in unsightly wrapping:

>   Example A.1. Examples of urlbase/cookiepath pairs for sharing login
>   cookies
>
>           urlbase is http://bugzilla.mozilla.org/
>           cookiepath is /
>           urlbase is http://tools.mysite.tld/bugzilla/
>                   but you have http://tools.mysite.tld/someotherapp/ whic
>   h shares
>                   authentication with your Bugzilla
>           cookiepath is /

Comment 5

[Frédéric Buclin]

(In reply to Byron Jones ‹:glob› from comment #4)
> the documentation in en/README.doc needs to be updated to include the
> "xmlto" and "dblatex" pacakges, as these are new dependencies

Yeah, I planned to do this in a separate patch, once the code changes were first approved. I can include these changes in the same patch too.


> the pdf now has colour, which i think we should avoid

I think colours are an improvement. It makes much clearer what is clickable. I don't want to remove them.


> on the contents are red, links are purple).  making everything black on
> white on the pdf will make it more printer friendly.

You are free to print the document in black, if you don't want colors.



> the sample for code on page 3 is corrupt:
>   %%%%Thisisfile‘utf8x.def’,%%generatedwiththedocstrip
>   </para>
> it should look like:
>   <para> Beginning and end of paragraph </para>

... assuming I can fix that. I will have to look at this problem.


> text in lists (eg. sections 5.3 and 5.5.4) isn't wrapped, and extends page
> the right margin.

I know. This is a problem with <simplelist>. This problem already exists in the current PDF file, so this is not a regression. I plan to fix this problem in a separate bug. This patch is already long enough.


> the link colour is almost identical to the body text colour

I didn't change anything related to the link color, so this is not a regression AFAIK.


> code samples are no longer indented, which looks weird:

AFAIK, this is already the case in the current text file, so not a regression.


So I will investigate the corruption you are talking about above, but I don't plan to fix existing bugs which are not regressions here. Let's first move to xmlto + dblatex, and then focus on problems which are already present for years (I was going to say "forever").

Comment 6

[:glob ✱]

(In reply to Frédéric Buclin from comment #5)
> > the pdf now has colour, which i think we should avoid
> 
> I think colours are an improvement. It makes much clearer what is clickable.
> I don't want to remove them.

my print driver doesn't have a "force black & white" mode, so i end up with light gray (unreadable) for coloured text.

nevertheless, i agree it does look much better on screen, so let's keep the colour.

> So I will investigate the corruption you are talking about above, but I
> don't plan to fix existing bugs which are not regressions here. Let's first
> move to xmlto + dblatex, and then focus on problems which are already
> present for years (I was going to say "forever").

ok :)

Comment 7

[Frédéric Buclin]

Attachment: patch, v2

This fixes the documentation in README.docs, which is now much shorter as all the hacks are no longer needed. This also fixes the problem with %%%%Thisisfile‘utf8x.def’,%%generatedwiththedocstrip. The problem was that pdflatex seems unable to correctly render <programlisting> within <informaltable>. As the UI was ugly in all cases, I removed the table entirely and I think the new UI is better. As I said, other problems are not regressions and will be fixed in subsequent bugs.

Comment 8

[:glob ✱]

Comment on attachment 567143 [details] [diff] [review]
patch, v2

sorry, i found a few more issues i missed the first time:


any urls in the footnote which contain ampersands are incorrect:

> http://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla\&component=Documentation

note the \ before the &


some footnotes in the glossary are missing -- only #6 and #10 are visible.

Comment 9

[Frédéric Buclin]

Attachment: patch, v3

This patch changes the color of all links to blue instead of the default red + magenta colors, which are harder to read. It also fixes the problem of <prompt> and <command> not being displayed in <screen></screen>.

About & being displayed as \& in the footnotes, this seems to be a bug in TeX 2007. I cannot reproduce with TeX 2010. Moreover, this only affects the URL in the footnotes, not the linkified text displayed inline. So if you click on this link, everything will work correctly. So this is a minor annoyance we can live with till RHEL has TeX 2010, especially when you know this affects only 2 links in 101 pages.

About footnotes not being displayed in the glossary, this is due to a limitation of dblatex as explained here: http://dblatex.sourceforge.net/doc/manual/ulink.footnotes.html (see the Dblatex Limitation section). As the text itself is still linkified and clickable, this is not a major problem.

Comment 10

[:glob ✱]

(In reply to Frédéric Buclin from comment #9)
> About & being displayed as \& in the footnotes, this seems to be a bug in TeX 2007.

i agree; however the mozilla systems which generates the official pdf are running an operating system where tex 2010 is not yet available.

> ... especially when you know this affects only 2 links in 101 pages

the amount of times something is broken isn't important; i only care if it's > 0.

> As the text itself is still linkified and clickable, this is not a major problem.

i disagree. i see the main point of the pdf documentation is to produce an easily printable form.  if someone wanted to view the documentation online with working links, they can use the fully functional html version.

with that in mind, the urls need to be always displayed, and always correct.

if we don't care about the printed form then i nominate that we drop the pdf completely.

Comment 11

[Frédéric Buclin]

Attachment: patch, v4

I replaced & by ; as TeX 2007 escapes ampersands incorrectly.

Comment 12

[:glob ✱]

Comment on attachment 568000 [details] [diff] [review]
patch, v4

r=glob

thanks for sorting out those issues!

Comment 13

[Max Kanat-Alexander]

This is so amazing, thank you! :-)

Have you guys checked that the API docs still look right with those stylesheet changes? I believe previously the stylesheet was being used only for the API docs.

Comment 14

[Max Kanat-Alexander]

Comment on attachment 568000 [details] [diff] [review]
patch, v4

Review of attachment 568000 [details] [diff] [review]:
-----------------------------------------------------------------

::: docs/bugzilla.ent.tmpl
@@ +3,5 @@
> +<!ENTITY current-year "2011">
> +
> +<!ENTITY min-perl-ver "5.8.1">
> +<!ENTITY landfillbase "http://landfill.bugzilla.org/bugzilla-tip/">
> +<!ENTITY bzg-bugs "<ulink url='http://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla;component=Documentation'>Bugzilla Documentation</ulink>">

That text is now untranslatable, no? Perhaps just the link should be the entity.

Comment 15

[Frédéric Buclin]

Attachment: patch, v5

(In reply to Max Kanat-Alexander from comment #13)
> Have you guys checked that the API docs still look right with those
> stylesheet changes? I believe previously the stylesheet was being used only
> for the API docs.

The stylesheet is used both by the API docs and the HTML docs. I just reverted one change to leave the background color intact for <code class="code"> in the API docs. So I left .code {} alone and duplicated what I needed into pre.code {} so that the HTML pages look nice too. :)


(In reply to Max Kanat-Alexander from comment #14)
> That text is now untranslatable, no? Perhaps just the link should be the
> entity.

Good point. Fixed!


Carrying forward glob's r+.

Comment 16

[Frédéric Buclin]

Committing to: bzr+ssh://[e-mail]/bugzilla/trunk/
added docs/bugzilla.ent.tmpl
modified docs/makedocs.pl
modified docs/style.css
added docs/xsl
modified docs/en/README.docs
modified docs/en/xml/Bugzilla-Guide.xml
modified docs/en/xml/about.xml
modified docs/en/xml/administration.xml
modified docs/en/xml/conventions.xml
modified docs/en/xml/customization.xml
modified docs/en/xml/gfdl.xml
modified docs/en/xml/glossary.xml
modified docs/en/xml/installation.xml
modified docs/en/xml/modules.xml
modified docs/en/xml/patches.xml
modified docs/en/xml/security.xml
modified docs/en/xml/troubleshooting.xml
modified docs/en/xml/using.xml
added docs/xsl/bugzilla-docs.xsl
added docs/xsl/chunks.xsl
added docs/xsl/nochunks.xsl
added docs/xsl/pdf.xsl
Committed revision 7993.

Comment 17

[Frédéric Buclin]

Added to relnotes for 4.4.