Bug 1535452 Comment 1 Edit History

Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.

This comment is a description of how the doc system works with some links to code. This is a very general overview that is intended to provide context for anyone working on any of the dependencies filed against this meta bug.


## Registration

The goal of our doc system is to allow individual teams and developers the ability to create and maintain their own domain specific documentation. We don't want to act as gatekeepers. In order for this to work, we need some mechanism for developers to register a doc dir.

Documentation gets registered by developers who wish to include their documentation via `moz.build` files. For example:
https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/moz.build#53

There are two variables, `SPHINX_TREES` and `SPHINX_PYTHON_PACKAGE_DIRS` and some information on them can be found here:
https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/python/mozbuild/mozbuild/frontend/context.py#2004


## Generation

Documentation is generated using [sphinx](http://www.sphinx-doc.org/en/master/) with some custom tooling to hook in all of the directories that are registered via `moz.build`. Aside from sphinx itself, there are 3 pieces to this stage:

1. [mach command](https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/docs/mach_commands.py#54)
This is command line entry point for generating and uploading docs. It is mostly just a thin wrapper around `sphinx-build`, but also acts like the glue the holds the system together.

2. [custom sphinx extension](https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/sphinx.py)
A custom sphinx extension that acts as a hook from which we can run all our customized operations before the regular 'sphinx-build' kicks in. This module is registered in the main [conf.py](https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/docs/conf.py#41). This extension adds a couple custom sphinx directives as well, but otherwise isn't very interesting.

3. [moztreedocs](https://searchfox.org/mozilla-central/source/tools/docs/moztreedocs/__init__.py)
A little python module containing the meat of the custom generation code. It is invoked from `sphinx.py` above and is responsible for reading the `moz.build` variables and copying all of the relevant source files into a staging directory in the objdir. Once all the source files have been copied, `sphinx-build` will run against the staging dir.

At a high level, the doc generation process will go something like this:

1. User runs `mach doc` or `mach doc <subdir>`
2. Mach command runs `sphinx-build` with `tools/docs/conf.py` as the config file.
3. The `mozbuild.sphinx` extension is registered in this config file.
4. This extension has a setup hook which calls into `moztreedocs`.
5. `moztreedocs` reads all the `moz.build` sphinx variables and copies all the source files to a staging dir.
6. The `mozbuild.sphinx` extension gets `sphinx-build` to build the docs from this staging directory.
7. `sphinx-build` resumes as normal, typically a directory next to the staging dir will contain the html output.


## Publication

Publishing docs is a separate step from generation, but uses the same mach command: `mach doc --upload`. This step is much simpler than the generation one. The docs are hosted on a static Amazon S3 bucket behind this domain:
https://firefox-source-docs.mozilla.org/

The `mach doc --upload` command uses Amazon's `boto3` library to push the output dir generated in step 7 above up to the static server:
https://searchfox.org/mozilla-central/source/tools/docs/moztreedocs/upload.py

And that's it.


## Continuous Integration

While it is possible to generate docs locally, publishing requires special credentials. Generation and publication are handled by two separate tasks (`Doc` and `DocUp` respectively). They are defined here:
https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/doc.yml

These tasks will run whenever someone modifies a doc source file, so changes to the documentation should be automatically picked up and published once the patch lands to mozilla-central. While it's nice to have this automated system, we do still care about running `mach doc` locally as developers will typically want to see how their modifications look before pushing.
This comment is a description of how the doc system works with some links to code. This is a very general overview that is intended to provide context for anyone working on any of the dependencies filed against this meta bug.


## Registration

The goal of our doc system is to allow individual teams and developers the ability to create and maintain their own domain specific documentation. We don't want to act as gatekeepers. In order for this to work, we need some mechanism for developers to register a doc dir.

Documentation gets registered by developers via `moz.build` files. For example:
https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/moz.build#53

There are two variables, `SPHINX_TREES` and `SPHINX_PYTHON_PACKAGE_DIRS` and some information on them can be found here:
https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/python/mozbuild/mozbuild/frontend/context.py#2004


## Generation

Documentation is generated using [sphinx](http://www.sphinx-doc.org/en/master/) with some custom tooling to hook in all of the directories that are registered via `moz.build`. Aside from sphinx itself, there are 3 pieces to this stage:

1. [mach command](https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/docs/mach_commands.py#54)
This is command line entry point for generating and uploading docs. It is mostly just a thin wrapper around `sphinx-build`, but also acts like the glue the holds the system together.

2. [custom sphinx extension](https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/sphinx.py)
A custom sphinx extension that acts as a hook from which we can run all our customized operations before the regular 'sphinx-build' kicks in. This module is registered in the main [conf.py](https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/docs/conf.py#41). This extension adds a couple custom sphinx directives as well, but otherwise isn't very interesting.

3. [moztreedocs](https://searchfox.org/mozilla-central/source/tools/docs/moztreedocs/__init__.py)
A little python module containing the meat of the custom generation code. It is invoked from `sphinx.py` above and is responsible for reading the `moz.build` variables and copying all of the relevant source files into a staging directory in the objdir. Once all the source files have been copied, `sphinx-build` will run against the staging dir.

At a high level, the doc generation process will go something like this:

1. User runs `mach doc` or `mach doc <subdir>`
2. Mach command runs `sphinx-build` with `tools/docs/conf.py` as the config file.
3. The `mozbuild.sphinx` extension is registered in this config file.
4. This extension has a setup hook which calls into `moztreedocs`.
5. `moztreedocs` reads all the `moz.build` sphinx variables and copies all the source files to a staging dir.
6. The `mozbuild.sphinx` extension gets `sphinx-build` to build the docs from this staging directory.
7. `sphinx-build` resumes as normal, typically a directory next to the staging dir will contain the html output.


## Publication

Publishing docs is a separate step from generation, but uses the same mach command: `mach doc --upload`. This step is much simpler than the generation one. The docs are hosted on a static Amazon S3 bucket behind this domain:
https://firefox-source-docs.mozilla.org/

The `mach doc --upload` command uses Amazon's `boto3` library to push the output dir generated in step 7 above up to the static server:
https://searchfox.org/mozilla-central/source/tools/docs/moztreedocs/upload.py

And that's it.


## Continuous Integration

While it is possible to generate docs locally, publishing requires special credentials. Generation and publication are handled by two separate tasks (`Doc` and `DocUp` respectively). They are defined here:
https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/doc.yml

These tasks will run whenever someone modifies a doc source file, so changes to the documentation should be automatically picked up and published once the patch lands to mozilla-central. While it's nice to have this automated system, we do still care about running `mach doc` locally as developers will typically want to see how their modifications look before pushing.
This comment is a description of how the doc system works with some links to code. This is a very general overview that is intended to provide context for anyone working on any of the dependencies filed against this meta bug.


## Registration

The goal of our doc system is to allow individual teams and developers the ability to create and maintain their own domain specific documentation. We don't want to act as gatekeepers. In order for this to work, we need some mechanism for developers to register a doc dir.

Documentation gets registered by developers via `moz.build` files. For example:
https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/moz.build#53

There are two variables, `SPHINX_TREES` and `SPHINX_PYTHON_PACKAGE_DIRS` and some information on them can be found here:
https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/python/mozbuild/mozbuild/frontend/context.py#2004


## Generation

Documentation is generated using [sphinx](http://www.sphinx-doc.org/en/master/) with some custom tooling to hook in all of the directories that are registered via `moz.build`. Aside from sphinx itself, there are 3 pieces to this stage:

1. [mach command](https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/docs/mach_commands.py#54)
This is the command line entry point for generating and uploading docs. It is mostly just a thin wrapper around `sphinx-build`, but also acts like the glue that holds the system together.

2. [custom sphinx extension](https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/sphinx.py)
A custom sphinx extension that acts as a hook from which we can run all our customized operations before the regular 'sphinx-build' kicks in. This module is registered in the main [conf.py](https://searchfox.org/mozilla-central/rev/4763b8d576ce52625d245d1ab6d9404ea025b026/tools/docs/conf.py#41). This extension adds a couple custom sphinx directives as well, but otherwise isn't very interesting.

3. [moztreedocs](https://searchfox.org/mozilla-central/source/tools/docs/moztreedocs/__init__.py)
A little python module containing the meat of the custom generation code. It is invoked from `sphinx.py` above and is responsible for reading the `moz.build` variables and copying all of the relevant source files into a staging directory in the objdir. Once all the source files have been copied, `sphinx-build` will run against the staging dir.

At a high level, the doc generation process will go something like this:

1. User runs `mach doc` or `mach doc <subdir>`
2. Mach command runs `sphinx-build` with `tools/docs/conf.py` as the config file.
3. The `mozbuild.sphinx` extension is registered in this config file.
4. This extension has a setup hook which calls into `moztreedocs`.
5. `moztreedocs` reads all the `moz.build` sphinx variables and copies all the source files to a staging dir.
6. The `mozbuild.sphinx` extension gets `sphinx-build` to build the docs from this staging directory.
7. `sphinx-build` resumes as normal, typically a directory next to the staging dir will contain the html output.


## Publication

Publishing docs is a separate step from generation, but uses the same mach command: `mach doc --upload`. This step is much simpler than the generation one. The docs are hosted on a static Amazon S3 bucket behind this domain:
https://firefox-source-docs.mozilla.org/

The `mach doc --upload` command uses Amazon's `boto3` library to push the output dir generated in step 7 above up to the static server:
https://searchfox.org/mozilla-central/source/tools/docs/moztreedocs/upload.py

And that's it.


## Continuous Integration

While it is possible to generate docs locally, publishing requires special credentials. Generation and publication are handled by two separate tasks (`Doc` and `DocUp` respectively). They are defined here:
https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/doc.yml

These tasks will run whenever someone modifies a doc source file, so changes to the documentation should be automatically picked up and published once the patch lands to mozilla-central. While it's nice to have this automated system, we do still care about running `mach doc` locally as developers will typically want to see how their modifications look before pushing.

Back to Bug 1535452 Comment 1