1245170 - Upload rust tooltool builds from taskcluster

Status: RESOLVED FIXED

(Taskcluster Graveyard :: Docker Images, defect)

Description

[Ralph Giles (:rillian)]

I'm experimenting with building rust toolchains on taskcluster and uploading to tooltool in taskcluster.

Comment 1

[Ralph Giles (:rillian)]

WIP on the `tc` branch of https://github.com/rillian/rust-build

Comment 2

[Ralph Giles (:rillian)]

[worker@0072f46032ee ~]$ /build/tooltool.py upload --message="Bug 1245170 - dummy file uploaded from taskcluster" -v --url=http://relengapi/tooltool/
DEBUG - processing 'upload' command with args ''
DEBUG - using options: {'cache_folder': None, 'algorithm': 'sha512', 'loglevel': 10, 'region': None, 'base_url': ['http://relengapi/tooltool/'], 'visibility': 
None, 'manifest': 'manifest.tt', 'auth_file': None, 'message': 'Bug 1245170 - dummy file uploaded from taskcluster', 'overwrite': False, 'size': 0.0}
DEBUG - materialized __main__.FileRecord(filename='test.txt', size=38, digest='c2b24b05462dcf465741cb3239826403a7be6d4fd8d996a02eedaf85e90faf07a522785c1feefa45
13bfac41a5143bd0a76716531f88b6a8c8d7cd2be780e197', algorithm='sha512', visibility=u'public')
DEBUG - loaded manifest from file 'manifest.tt'
DEBUG - hashed u'test.txt' with sha512 to be c2b24b05462dcf465741cb3239826403a7be6d4fd8d996a02eedaf85e90faf07a522785c1feefa4513bfac41a5143bd0a76716531f88b6a8c8
d7cd2be780e197
ERROR - Bad Request: Could not determine authenticated username

Comment 3

[Ralph Giles (:rillian)]

Per Dustin on irc, this comes from https://github.com/mozilla/build-relengapi/blob/aabf2d3ebe390cfb005d594a77366711aa97e816/relengapi/blueprints/tooltool/__init__.py#L136

We need to register a client for the tc relengapi-proxy to use.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: https://github.com/mozilla/build-relengapi/pull/361

Here's one possible fix here (certainly, landing this will unblock rillian).

We might be able to do better, perhaps by including some kind of identifier in the temporary token.  The issue there is, how can you trust that identifier?

Comment 5

[Mike Hommey [:glandium]]

(In reply to Ralph Giles (:rillian) from comment #1)
> WIP on the `tc` branch of https://github.com/rillian/rust-build

Please consider landing this on m-c once it works. I've done similar work to build GCC on TC through pushes to try.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8715018 [details] [review]
https://github.com/mozilla/build-relengapi/pull/361

So, the best I can think of with the tokens is for the issuance of a temporary token to include the issuing token's identity as a claim.  The downside is, the identity of the issuing token here is 'tid:1234' for some token-id, so that actually doesn't say much.  To be honest, the TC temporary-token model is already better and getting even better now, so I'd rather switch to using that model than continue to iterate on improving the relengapi model.

I've updated the PR with tests.

Comment 7

[Ralph Giles (:rillian)]

(In reply to Mike Hommey [:glandium] from comment #5)

> Please consider landing this on m-c once it works.

Sure. The previous iteration landed in bug 1237366.

> I've done similar work to build GCC on TC through pushes to try.

Where is this in the tree? I wasn't able to find it.

Comment 8

[Mike Hommey [:glandium]]

(In reply to Ralph Giles (:rillian) from comment #7)
> > I've done similar work to build GCC on TC through pushes to try.
> 
> Where is this in the tree? I wasn't able to find it.

I still need to figure how to actually hook it up with the try syntax.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

OK, the fix to tooltool in comment 4 is landed now.

Comment 10

[Ralph Giles (:rillian)]

Confirming that we can talk to tooltool from taskcluster jobs now.

Next issue is that python's urllib2 can't talk to s3 (or any other https location). I get an SSL certificate validation error. I believe this is because on centos 5 the cert bundle is patched into the openssl package, which we remove, preferring our own build of a recent release from source. At least that's the theory. The image does have a /etc/pki/tls/cert.pem provided by openssl-0.9.8e-37.el5_11, and curl works fine. Maybe it got reinstalled after the python build? Investigating further.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

In the TaskCluster world, the more common approach to this sort of thing is to make the outputs artifacts of a task, and then use the index to find that task in every subsequent task where it's required.

We went down the upload-to-tooltool route instead thinking that it might be easier, but it's turning out it's not easier.  I wonder if we should re-target this to the artifact approach?

Comment 12

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

(In reply to Dustin J. Mitchell [:dustin] from comment #11)
> In the TaskCluster world, the more common approach to this sort of thing is
> to make the outputs artifacts of a task, and then use the index to find that
> task in every subsequent task where it's required.
> 
> We went down the upload-to-tooltool route instead thinking that it might be
> easier, but it's turning out it's not easier.  I wonder if we should
> re-target this to the artifact approach?

I cannot do this because I need to support buildbot-based Android builds.  The runway to doing this all in TC is about a million miles long :(

Comment 13

[Mike Hommey [:glandium]]

Note I'd like to do something similar for gcc, clang and gtk3 packages, which also need to be used by both buildbot and TC builds. Tooltool is still the preferred way.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, fair enough.  So, we can continue to whack away at the Python/SSL issues, I suppose.

Comment 15

[Ralph Giles (:rillian)]

(In reply to Dustin J. Mitchell [:dustin] from comment #11)

> In the TaskCluster world, the more common approach to this sort of thing is
> to make the outputs artifacts of a task, and then use the index to find that
> task in every subsequent task where it's required.

Could we make this work for buildbot by extending the tooltool client so it can pull artefacts from taskcluster?

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

That feels pretty dirty to me, and furthermore I've found that making substantial changes to the tooltool client in the Gecko codebase is a nightmare all its own, as there are a half-dozen different ways of running tooltool.

Comment 17

[Ralph Giles (:rillian)]

As an update, I've been able to upload test files from taskcluster. I'm continuing to work out the automation bugs for actual rust builds, but the cycle time is long. I'll tag and import the new scripts once I've confirmed it's working.

Comment 18

[Ralph Giles (:rillian)]

Attachment: Move tooltool upload container in tree

I've tagged a version confirmed working. This built and uploaded a linux toolchain to tooltool entirely within taskcluster:

[
{
"size": 89700424,
"visibility": "public",
"digest": "84828e448fcf0af17f7a6cb26a75838aa20feb0d20d0a33ef8fcda2f9d1428cac01b7ef85be271501089f49dd928b42b50c7f4f80afc0dd38205899e8fae223f",
"algorithm": "sha512",
"filename": "rustc.tar.xz"
}
]

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8719573 [details] [diff] [review]
Move tooltool upload container in tree

Review of attachment 8719573 [details] [diff] [review]:
-----------------------------------------------------------------

A few notes, but nothing to object to here -- land as-is or address the notes, as you see fit.

::: testing/docker/rust-build/tcbuild.py
@@ +1,5 @@
> +#!/bin/env python
> +'''
> +This script triggers a taskcluster task, waits for it to finish,
> +fetches the artifacts, uploads them to tooltool, and updates
> +the in-tree tooltool manifests.

I don't see this script referenced elsewhere -- is it necessary?

::: testing/docker/rust-build/upload_rust.sh
@@ +3,5 @@
> +set -x -e
> +
> +: WORKSPACE ${WORKSPACE:=/home/worker}
> +
> +CORES=$(nproc || grep -c ^processor /proc/cpuinfo || sysctl -n hw.ncpu)

You don't need this

@@ +18,5 @@
> +  MESSAGE="Rust toolchain build for gecko"
> +fi
> +if test -r rust-version; then
> +  MESSAGE="$MESSAGE $(cat rust-version)"
> +fi

Thanks for the nice message construction!!

@@ +19,5 @@
> +fi
> +if test -r rust-version; then
> +  MESSAGE="$MESSAGE $(cat rust-version)"
> +fi
> +/build/tooltool.py upload ${TOOLTOOL_OPTS} --message="${MESSAGE}"

Just to check: at the completion of the task, this file may not yet be available for a delay on the order of minutes -- there are crontasks that process the tooltool uploads that do not run immediately.  Is that OK?

Comment 20

[Ralph Giles (:rillian)]

(In reply to Dustin J. Mitchell [:dustin] from comment #19)

> > +This script triggers a taskcluster task, waits for it to finish,
> > +fetches the artifacts, uploads them to tooltool, and updates
> > +the in-tree tooltool manifests.
> 
> I don't see this script referenced elsewhere -- is it necessary?

It's the way I trigger builds currently, so I think it's important to have in-tree, and it's necessary until I figure out how to get this hooked up to try pushes.


> > +CORES=$(nproc || grep -c ^processor /proc/cpuinfo || sysctl -n hw.ncpu)
> 
> You don't need this

Right. This was cut-and-pasted from the build script. I've removed it upstream so it can be dropped in the next update.

> Just to check: at the completion of the task, this file may not yet be
> available for a delay on the order of minutes -- there are crontasks that
> process the tooltool uploads that do not run immediately.  Is that OK?

I noticed there was a delay when I was verifying the test runs. I suspect it will be ok: job dispatch can take several minutes too, and the harness which calls tooltool retries for a while if a resource isn't available.

Comment 21

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d15669bc8eef

Comment 22

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d15669bc8eef