Closed Bug 1405408 Opened 7 years ago Closed 7 years ago

Add support for private toolchains

Categories

(Taskcluster :: General, enhancement)

Product:
Taskcluster
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla58

People

(Reporter: nalexander, Assigned: nalexander)

References

Details

Attachments

(5 files)

Bug 1405408 - Pre: Handle query string and fragments in URL parameters.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1405408 - Part 0: Let toolchain tasks create non-public/build artifacts.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1405408 - Part 1: Use Task Cluster proxy to download non-public artifacts.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1405408 - Part 2: Add taskcluster-proxy and scopes to tasks consuming non-public/ toolchains.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1405408 - Post: Use taskcluster-proxy /bewit endpoint to download private toolchains.
59 bytes, text/x-review-board-request
dustin
: review+
Details 
I'd like to migrate the Android SDK to a toolchain task, but I believe it's not legal to publicly redistribute the SDK.  (Google makes one agree to a license during installation.)

The existing Deps task produces a private Task Cluster artifact:
see http://searchfox.org/mozilla-central/rev/a4702203522745baff21e519035b6c946b7d710d/taskcluster/scripts/builder/build-android-dependencies/after.sh#18-20.

This ticket tracks allowing private Task Cluster artifacts to be produced by toolchain tasks (which might just work) and to be consumed by tasks in automation (which might require additional permissions, we shall see).
On IRC, glandium and I briefly discussed this:

[2017-08-16 18:03:24] <nalexander> froydnj: ted: looks like you figured things out.  There's a ticket (I think?) for making the whole process a bunch of toolchain tasks, but it's not doable yet for private artifacts.
[2017-08-16 18:03:48] <glandium> nalexander: I don't see why not
[2017-08-16 18:04:09] <nalexander> glandium: sorry, no technical blocker -- just that it's not in place for private artifacts yet.
[2017-08-16 18:04:20] <nalexander> glandium: please make it happen, and I can move that stuff to toolchain tasks :)
[2017-08-16 18:04:40] <glandium> nalexander: I think the only thing missing is giving the tasks the scope to download private artifacts
[2017-08-16 18:05:04] <nalexander> glandium: that would be great.
[2017-08-16 18:05:15] <glandium> that is, the tasks that use the artifacts
[2017-08-16 18:05:43] <glandium> or maybe they already have those scopes by default
[2017-08-16 18:06:23] <glandium> looks like not
[2017-08-16 18:06:29] <nalexander> glandium: I doubt they have those scopes by default; I saw a list a while back.

dustin: glandium: do either of you know where the scopes are set?  In the meantime, I'll try this and see it if Just Works.
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(dustin)
Blocks: 1405412
We can assign scopes to SCM repositories by level.

https://docs.taskcluster.net/manual/design/namespaces#artifacts gives some information on naming artifacts.  We've typically used project/gecko for gecko stuff that is not repo-specific, so that's a good choice.
Flags: needinfo?(dustin)
Depends on: 1405889
Blocks: 1270667
Comment on attachment 8916128 [details]
Bug 1405408 - Pre: Handle query string and fragments in URL parameters.

https://reviewboard.mozilla.org/r/187366/#review192800

I'm not sure I'm the right r? here, but it's simple enough.
Attachment #8916128 - Flags: review?(dustin) → review+
Comment on attachment 8916129 [details]
Bug 1405408 - Part 0: Let toolchain tasks create non-public/build artifacts.

https://reviewboard.mozilla.org/r/187368/#review192802
Attachment #8916129 - Flags: review?(dustin) → review+
Comment on attachment 8916130 [details]
Bug 1405408 - Part 1: Use Task Cluster proxy to download non-public artifacts.

https://reviewboard.mozilla.org/r/187370/#review192806

This won't work on the command line -- is that OK?
Attachment #8916130 - Flags: review?(dustin) → review+
Comment on attachment 8916131 [details]
Bug 1405408 - Part 2: Add taskcluster-proxy and scopes to tasks consuming non-public/ toolchains.

https://reviewboard.mozilla.org/r/187372/#review192808

::: taskcluster/taskgraph/transforms/use_toolchains.py:104
(Diff revision 1)
> +                # For example, add 'scopes: [queue:get-artifact:path/to/*]'
> +                # for 'path/to/artifact.tar.xz'.
> +                job['worker']['taskcluster-proxy'] = True
> +                scope = 'queue:get-artifact:{}/*'.format(dirname)
> +                if scope not in scopes:
> +                    scopes.append(scope)

:thumbsup:
Attachment #8916131 - Flags: review?(dustin) → review+
Comment on attachment 8916132 [details]
Bug 1405408 - Post: Use taskcluster-proxy /bewit endpoint to download private toolchains.

https://reviewboard.mozilla.org/r/187374/#review192810
Attachment #8916132 - Flags: review?(dustin) → review+
Flags: needinfo?(mh+mozilla)
Comment on attachment 8916130 [details]
Bug 1405408 - Part 1: Use Task Cluster proxy to download non-public artifacts.

https://reviewboard.mozilla.org/r/187370/#review192806

My take is that yes, it's OK.  I don't really understand what could work on the command line, but it should be a follow-up ticket.  (If you or glandium have thoughts, please file now and capture them.)
(In reply to Dustin J. Mitchell [:dustin] from comment #14)
> Comment on attachment 8916132 [details]
> Bug 1405408 - Post: Use taskcluster-proxy /bewit endpoint to download
> private toolchains.
> 
> https://reviewboard.mozilla.org/r/187374/#review192810

jonasf: Bug 1405889 isn't marked fixed, and even when it is the proxy still won't stream correctly.  I've worked around it by using the /bewit endpoint (in the patch linked above).  Are you okay using the /bewit endpoint instead of the (broken) pass through for now?  For ever?  Or do you think it's important to _not_ use the /bewit endpoint in production for some reason?
Flags: needinfo?(jopsen)
(In reply to Nick Alexander :nalexander from comment #16)
> (In reply to Dustin J. Mitchell [:dustin] from comment #14)
> > Comment on attachment 8916132 [details]
> > Bug 1405408 - Post: Use taskcluster-proxy /bewit endpoint to download
> > private toolchains.
> > 
> > https://reviewboard.mozilla.org/r/187374/#review192810
> 
> jonasf: Bug 1405889 isn't marked fixed, and even when it is the proxy still
> won't stream correctly.  I've worked around it by using the /bewit endpoint
> (in the patch linked above).  Are you okay using the /bewit endpoint instead
> of the (broken) pass through for now?  For ever?  Or do you think it's
> important to _not_ use the /bewit endpoint in production for some reason?

garndt: just saw you update Bug 1405889 with deployment information, thanks!  Do you have an opinion on this question?
Flags: needinfo?(garndt)
Hrm, I do not have a strong opinion but what does come to mind is URLs that are leaked into the logs that contain short lived bewit tokens.  In the case of these artifacts, that might be ok, because technically we're not redistributing the SDK intentionally, however, if people use this for artifacts that are truly private, then that's a different story.

Maybe it's a non-issue.
Flags: needinfo?(garndt)
> Or do you think it's important to _not_ use the /bewit endpoint in production for some reason?

I don't have any strong objection, but as garndt says try to not leak the bewit URLs (these are effectively signed URLs).
So if the resource they are fetching is private for a good reason, then I guess not dumping them to log is a good idea.
Flags: needinfo?(jopsen)
(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #19)
> > Or do you think it's important to _not_ use the /bewit endpoint in production for some reason?
> 
> I don't have any strong objection, but as garndt says try to not leak the
> bewit URLs (these are effectively signed URLs).
> So if the resource they are fetching is private for a good reason, then I
> guess not dumping them to log is a good idea.

Thanks, folks!  I did dump some of these /bewit endpoints while debugging, but I'm quite confident they're not leaked to logs at this time.  I'll verify before landing, and then bombs away!
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/831c287a4ae8
Pre: Handle query string and fragments in URL parameters. r=dustin
https://hg.mozilla.org/integration/autoland/rev/347192dde0ed
Part 0: Let toolchain tasks create non-public/build artifacts. r=dustin
https://hg.mozilla.org/integration/autoland/rev/74e5d77b144d
Part 1: Use Task Cluster proxy to download non-public artifacts. r=dustin
https://hg.mozilla.org/integration/autoland/rev/20cae3b446bf
Part 2: Add taskcluster-proxy and scopes to tasks consuming non-public/ toolchains. r=dustin
https://hg.mozilla.org/integration/autoland/rev/5fa05ab4204d
Post: Use taskcluster-proxy /bewit endpoint to download private toolchains. r=dustin
Blocks: 1408458
Assignee: nobody → nalexander
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: