Closed Bug 1382729 Opened 7 years ago Closed 7 years ago

Publish Firefox's Sphinx docs from TaskCluster

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla56

People

(Reporter: gps, Assigned: dustin)

References

Details

Attachments

(3 files)

https://gecko.readthedocs.org/ frequently fails to build. We keep hitting apparent scaling limits with ReadTheDocs. Plus the setup is a bit fragile.

As of bug 1048650 we now generate the Sphinx docs as part of TaskCluster. So it makes sense to publish these generated docs somewhere and to tie that to TaskCluster.

Bug 877197 is on file for MDN to support publishing Sphinx docs. But with the recent pivot of MDN, my guess is that bug is going nowhere and will be WONTFIX'd.

We should come up with somewhere else to publish the Firefox Sphinx docs. Somewhere reliable. Preferably somewhere hosted or under control of Mozilla so we can fix it when it is broken.

Apparently you can publish TC artifacts to S3 buckets. You can also configure S3 buckets for web site publishing. We should be able to create an S3 bucket, do some DNS and x509 certificate magic, and have TC publish the Sphinx docs.

Dustin: is TC in a position to manage the S3 bucket? Or do I need to work the EC2 bits on my end?
Flags: needinfo?(dustin)
We can set up publishing to an S3 bucket.  It's a little fiddly on the backend, and has the restriction that the TC AWS account has to own the bucket, but otherwise is pretty straightforward.  We can also set up CloudFront to use custom DNS and TLS (if desired).  The uploading task basically calls a TC endpoint and gets back some AWS STS credentials which it can then use to upload.
Flags: needinfo?(dustin)
Well, since the TC AWS account has to own the bucket, I guess I'm off the hook for implementing this :) Seriously though, I can help with Sphinx updates if needed.

As far as prioritization, this likely isn't at the top of anyone's list.

That being said, MDN is effectively going away for build system docs. And sphinx-js is a nifty way to document JS which I'd like to explore but fear straining RTD. I suspect if we made the in-tree Sphinx docs easier to update and more useful, a lot of people would be very appreciative. Since RTD inconsistency is undermining trust in the in-tree Sphinx docs, having this implemented and having stable Sphinx docs generation would likely spur interest in improving docs.
OK, I'll get a task created that uploads a hello.html, and then let you take it from there.
Assignee: nobody → dustin
That division of labor is very acceptable to me!
Actually, https://docs.taskcluster.net/manual/using/s3-uploads means that any user can own the bucket.  I'll set it up with a TC bucket, but that's easy to change later.
I created bucket gecko-docs.mozilla.org, with the following bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadForGetBucketObjects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::gecko-docs.mozilla.org/*"
        }
    ]
}

And the result is

http://gecko-docs.mozilla.org.s3-website-us-west-2.amazonaws.com/

Once we figure out a hostname and get a TLS cert for it, we can set up cloudfront.  I have a bit more work to do to get credentials set up and run this in automation, then I'll post a patch.
The gecko-docs-upload IAM user has this policy (also named gecko-docs-upload) applied:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::gecko-docs.mozilla.org"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::gecko-docs.mozilla.org/*"
            ]
        }
    ]
}
I should add I didn't test the task invocation, so the requests/session stuff might be wrong.
Comment on attachment 8891500 [details]
Bug 1382729: Upload documentation to S3 automatically;

https://reviewboard.mozilla.org/r/162626/#review168020

::: tools/docs/mach_commands.py:113
(Diff revision 1)
> +    @Command('doc-upload', category='devenv',
> +        description='Generate and upload documentation from the tree.')

There are already too many mach commands. Please file a follow-up to consolidate `mach doc` into a sub-command based command.

::: tools/docs/mach_commands.py:126
(Diff revision 1)
> +    def upload_docs(self, bucket, region, what=None):
> +        self._activate_virtualenv()
> +        self.virtualenv_manager.install_pip_package('boto3==1.4.4')
> +
> +        outdir = os.path.join(self.topobjdir, 'docs')
> +        #self.build_docs(what=what, outdir=outdir, format='html')

I think you forgot to uncomment this :)

::: tools/docs/mach_commands.py:173
(Diff revision 1)
> +                    if content_type:
> +                        extra_args['ContentType'] = content_type
> +                    if content_encoding:
> +                        extra_args['ContentEncoding'] = content_encoding
> +                    print('uploading', pathname)
> +                    s3.upload_file(pathname, bucket, pathname, ExtraArgs=extra_args)

This will be much faster if you use a thread pool with futures like we do for the decision task. But it probably isn't worth doing initially.
Attachment #8891500 - Flags: review?(gps) → review+
Yeah, I only promised I'd get it working, remember? :)

A few other ideas for improvement:
 - list the bucket contents and only upload changed files
 - list the bucket contents and ensure that any objects already present are either present in the updated docs, or have a redirect defined (so that a link never dies)
I'll land this with the new job set to tier 3 so I can see it fail..
https://hg.mozilla.org/mozilla-central/rev/19f92c04608c
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
This is causing https://public-artifacts.taskcluster.net/bxhzSaMeT42advEOxbWW2w/0/public/logs/live_backing.log

[task 2017-07-31T15:45:04.609938Z] Creating task with taskId boA9lTVRSiSgsKsw-bo8ig for source-test-doc-upload
[task 2017-07-31T15:45:04.611541Z] Starting new HTTP connection (4): taskcluster
[task 2017-07-31T15:45:04.612161Z] Starting new HTTP connection (5): taskcluster
[task 2017-07-31T15:45:04.997049Z] "PUT /queue/v1/task/boA9lTVRSiSgsKsw-bo8ig HTTP/1.1" 403 11670
[task 2017-07-31T15:45:04.998205Z] You do not have sufficient scopes. This request requires you
[task 2017-07-31T15:45:04.998257Z] to have one of the following sets of scopes:
[task 2017-07-31T15:45:04.998362Z] [
[task 2017-07-31T15:45:04.998442Z]   [
[task 2017-07-31T15:45:04.998557Z]     "secrets:get:project/releng/gecko/build/level-3/gecko-docs-upload",
[task 2017-07-31T15:45:04.998683Z]     "secrets:get:project/taskcluster/gecko/hgfingerprint",
[task 2017-07-31T15:45:04.998741Z]     "docker-worker:cache:level-1-checkouts-v1",
[task 2017-07-31T15:45:04.998793Z]     "queue:route:tc-treeherder.v2.try.bcdd7a362ce11f0a20779c27d9b9d7261d853f2e.209370",
[task 2017-07-31T15:45:04.998872Z]     "queue:route:tc-treeherder-stage.v2.try.bcdd7a362ce11f0a20779c27d9b9d7261d853f2e.209370"
[task 2017-07-31T15:45:04.998895Z]   ]
[task 2017-07-31T15:45:04.998913Z] ]

on try.  I don't know why the job is running on try :(
backed out in

https://hg.mozilla.org/mozilla-central/rev/1be0c1da06076f85c69cd8a9d244e0164ec544d9

And ahal points out, run-on-projects doesn't apply to `-j`obs :(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Keywords: leave-open
Comment on attachment 8892987 [details]
Bug 1382729: add support for run-on-projects with -j;

https://reviewboard.mozilla.org/r/163990/#review169296

In my experimentation this increased the number of jobs filtered by target_tasks by one, and the source-test-doc-upload task did not appear in the output afterward.
Comment on attachment 8891500 [details]
Bug 1382729: Upload documentation to S3 automatically;

https://reviewboard.mozilla.org/r/162626/#review169300
Comment on attachment 8892987 [details]
Bug 1382729: add support for run-on-projects with -j;

https://reviewboard.mozilla.org/r/163990/#review169302

What could go wrong.
Attachment #8892987 - Flags: review?(gps) → review+
Well, I can imagine 10,000 things, so it's probably none of those :)

I ran a try job and it didn't fail, so that's a start!
Assignee: dustin → gps
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d2ad1e533a35
Upload documentation to S3 automatically; r=gps
https://hg.mozilla.org/integration/autoland/rev/f0df47303111
add support for run-on-projects with -j; r=gps
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/36ad88e6b7b2
Call proper mach command to uploads docs; r=dustin, a=bustage
Still busted on central with:

HTTPError: 404 Client Error: Not Found for url: http://taskcluster/secrets/repo:hg.mozilla.org/mozilla-central/gecko-docs-upload

It sounds like Dustin knows what's wrong.
Assignee: gps → dustin
Depends on: 1387498
I've set up gecko-docs.mozilla.org-{l1,l2} with 1-day expiration lifecycle rules, so the data won't stick around.

I also set up gecko-upload-docs-{l1,l2} IAM users with policies like

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::gecko-docs.mozilla.org-l1"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::gecko-docs.mozilla.org-l1/*"
            ]
        }
    ]
}

I set up secrets containing creds for those users.

Now to update the task.
Attached patch bug1382729.patchSplinter Review
This should fix things up.  This succeeded in try (finally): 
  https://tools.taskcluster.net/groups/N2XKmyNZTHSgyfstzhi7yA/tasks/XVevcptvS2C8fnOWMIDn5Q/runs/0/logs/public%2Flogs%2Flive.log
so i expect it will do OK at level 3 too.
Attachment #8895098 - Flags: review?(gps)
Attachment #8895098 - Flags: review?(gps) → review+
By giving Try tasks the ability to upload to an S3 bucket, we've just opened up the potential for hosting abuse. Also, if we screw things up, mistakes could persist forever. I assume there is a bucket policy with aggressive auto expiration to prevent bad things?
(In reply to Gregory Szorc [:gps] from comment #36)
> By giving Try tasks the ability to upload to an S3 bucket, we've just opened
> up the potential for hosting abuse. Also, if we screw things up, mistakes
> could persist forever. I assume there is a bucket policy with aggressive
> auto expiration to prevent bad things?

Yes, sorry I was thinking that discussion was in this bug.  It's a private bucket, without website support enabled, and it has a 1-day expiration set.  So no big risk.
Blocks: 1389341
Followup in other bugs..
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
Blocks: 1392765
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Product: TaskCluster → Firefox Build System
See Also: → 1539990
You need to log in before you can comment on or make changes to this bug.