1390693 - Add tags for different rtd versions

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Chenxia Liu [:liuche]]

Specifically on Mobile, some partners would like to have documentation for specific versions Firefox. Currently, we just show "latest".

What would it take to have versions of the docs tagged to each release?

Comment 1

[Gregory Szorc [:gps]]

Dustin: how should we do this in a TaskCluster world?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Maybe as part of the release process?  That could upload to a different prefix of the same bucket, or to a different bucket.

Comment 3

[Gregory Szorc [:gps]]

Path prefix in the bucket would be doable. We could put the central-based docs at / then have the beta and release repos deposit docs to /releases/<channel>/<version>/ or something. Last write wins. But that should be fine.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

You might also put a redirect from / to /latest or something, just to avoid one upload being a subdir of another (and avoiding the risk of an upload overwriting a release)

Comment 5

[Elvin Lee [:ellee] (no longer with Mozilla)]

(In reply to Chenxia Liu [:liuche] from comment #0)
> Specifically on Mobile, some partners would like to have documentation for
> specific versions Firefox. Currently, we just show "latest".

Just to clarify, I think it would be helpful to have documentation tagged by release for our own purposes. The ability to better respond to partner questions is an additional benefit. :)

Comment 6

[Gregory Szorc [:gps]]

Assuming we don't do any docs updates after central (which is probably fair because I can't think of any situation where we'd update docs on a release branch and not on central because to do would mean docs on central aren't correct), then docs generation on central can look at config/milestone.txt and upload to that path (stripping the "a1" so we have a straight version number).

Comment 7

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Run documentation generation everywhere;

We want to verify that Sphinx docs work on all repos. It is only
uploading that should be limited.

Review commit: https://reviewboard.mozilla.org/r/172322/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172322/

Comment 8

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Indent code to make linter happy;

I didn't fix all violations in this file. Something is better than
nothing.

Review commit: https://reviewboard.mozilla.org/r/172324/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172324/

Comment 9

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Use separate variable for the base output directory;

An upcoming commit will want to put something outside the
format-specific output directory.

Review commit: https://reviewboard.mozilla.org/r/172326/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172326/

Comment 10

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Generate docs archive within mach command;

By using mozpack, we get deterministic archives. This also makes the
task command simpler.

Review commit: https://reviewboard.mozilla.org/r/172328/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172328/

Comment 11

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Move S3 upload logic to standalone module;

We try to keep mach_commands.py files as minimal as possible.

Review commit: https://reviewboard.mozilla.org/r/172330/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172330/

Comment 12

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Use distribution_files() for finding files to upload;

Instead of doing the file finding inside s3_upload(), the function now
takes the output of distribution_files().

The new code is much simpler.

Review commit: https://reviewboard.mozilla.org/r/172332/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172332/

Comment 13

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Fold `mach doc-upload` into `mach doc`;

We now have an --upload flag to control whether upload is performed.

We don't inline it because we want to maintain a "firewall" between
regular docs and all the extra packages and imports needed for S3.

Review commit: https://reviewboard.mozilla.org/r/172334/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172334/

Comment 14

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Upload docs to project and version specific locations;

Previously, we uploaded the main Firefox tree docs to /.

In reality, there are multiple Sphinx projects in the repo. In
addition, it is sometimes desirable to access docs for an older
version of Firefox.

In this commit, we add support for specifying the S3 key prefix
for uploads. Then we change the upload code to upload to multiple
locations:

* <project>/latest (always)
* <project>/<version> (if a version is defined in the Sphinx config)
* / (for the main Sphinx docs project)

For the Firefox docs, ``version`` corresponds to a sanitized value from
``milestone.txt``. Currently, it resolves to ``57.0``.

While we're here, we add support for declaring an alternate project
name in the Sphinx conf.py file. If ``moz_project_name`` is defined,
we use that as the project name. For Firefox, we set it to ``main``.
This means our paths (local and uploaded) are now ``main`` instead of
``Mozilla_Source_Tree_Docs``. That's much more pleasant.

Review commit: https://reviewboard.mozilla.org/r/172336/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172336/

Comment 15

[Gregory Szorc [:gps]]

Attachment: Bug 1390693 - Use thread pool for S3 uploads;

This reduces the time taken to upload the Firefox docs from ~30s to
~5s (per invocation).

Review commit: https://reviewboard.mozilla.org/r/172338/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172338/

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900890 [details]
Bug 1390693 - Run documentation generation everywhere;

https://reviewboard.mozilla.org/r/172322/#review178072

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900891 [details]
Bug 1390693 - Indent code to make linter happy;

https://reviewboard.mozilla.org/r/172324/#review178074

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900892 [details]
Bug 1390693 - Use separate variable for the base output directory;

https://reviewboard.mozilla.org/r/172326/#review178076

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900893 [details]
Bug 1390693 - Generate docs archive within mach command;

https://reviewboard.mozilla.org/r/172328/#review178080

Comment 20

[Gregory Szorc [:gps]]

Comment on attachment 8900897 [details]
Bug 1390693 - Upload docs to project and version specific locations;

https://reviewboard.mozilla.org/r/172336/#review178082

::: tools/docs/mach_commands.py:156
(Diff revision 1)
> +        s3_upload(files, key_prefix='%s/latest' % project)
> +        if version:
> +            s3_upload(files, key_prefix='%s/%s' % (project, version))

We /may/ want to put the version first. Since the hostname is in the context of Firefox source tree docs, then "version" is a more important hierarchical component than "project."

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900894 [details]
Bug 1390693 - Move S3 upload logic to standalone module;

https://reviewboard.mozilla.org/r/172330/#review178084

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900895 [details]
Bug 1390693 - Use distribution_files() for finding files to upload;

https://reviewboard.mozilla.org/r/172332/#review178086

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900896 [details]
Bug 1390693 - Fold `mach doc-upload` into `mach doc`;

https://reviewboard.mozilla.org/r/172334/#review178088

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900898 [details]
Bug 1390693 - Use thread pool for S3 uploads;

https://reviewboard.mozilla.org/r/172338/#review178094

Comment 25

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8900897 [details]
Bug 1390693 - Upload docs to project and version specific locations;

https://reviewboard.mozilla.org/r/172336/#review178092

What will this leave at the root?

Note that you can upload "redirect" objects (empty objects with header website-redirect-location), so it might be good to link / -> /main and /main -> /main/latest or something like that.

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8900897 [details]
Bug 1390693 - Upload docs to project and version specific locations;

https://reviewboard.mozilla.org/r/172336/#review178092

The main docs will still be available at root.

Thinking about this more, I think we should land this now and deal with any refactoring to the hierarchy in a follow-up. (Why let good patches go to waste?) I'd like to retroactively generate docs for old versions, purge the S3 bucket, and start from a clean slate with the new naming schema, including support for redirects.

We can hash that out in another bug.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0180878e6feb
Run documentation generation everywhere; r=dustin
https://hg.mozilla.org/integration/autoland/rev/9fe542a58029
Indent code to make linter happy; r=dustin
https://hg.mozilla.org/integration/autoland/rev/85ae0d0798be
Use separate variable for the base output directory; r=dustin
https://hg.mozilla.org/integration/autoland/rev/c1d7fe990f07
Generate docs archive within mach command; r=dustin
https://hg.mozilla.org/integration/autoland/rev/ed802f889444
Move S3 upload logic to standalone module; r=dustin
https://hg.mozilla.org/integration/autoland/rev/c74c6ded8370
Use distribution_files() for finding files to upload; r=dustin
https://hg.mozilla.org/integration/autoland/rev/974718bfc596
Fold `mach doc-upload` into `mach doc`; r=dustin
https://hg.mozilla.org/integration/autoland/rev/c0b6cbecee95
Upload docs to project and version specific locations; r=dustin
https://hg.mozilla.org/integration/autoland/rev/7f3526c6c5d7
Use thread pool for S3 uploads; r=dustin

Comment 28

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/0180878e6feb
https://hg.mozilla.org/mozilla-central/rev/9fe542a58029
https://hg.mozilla.org/mozilla-central/rev/85ae0d0798be
https://hg.mozilla.org/mozilla-central/rev/c1d7fe990f07
https://hg.mozilla.org/mozilla-central/rev/ed802f889444
https://hg.mozilla.org/mozilla-central/rev/c74c6ded8370
https://hg.mozilla.org/mozilla-central/rev/974718bfc596
https://hg.mozilla.org/mozilla-central/rev/c0b6cbecee95
https://hg.mozilla.org/mozilla-central/rev/7f3526c6c5d7