Make use of scope-protected caches in Windows builds

NEW
Assigned to

Status

Taskcluster
Task Configuration
8 months ago
8 months ago

People

(Reporter: pmoore, Assigned: pmoore, Mentored)

Tracking

({good-first-bug})

Details

(Assignee)

Description

8 months ago
We currently have caches in our Windows builds, but they are not protected by scopes, rather any task that runs on a given worker, could potentially poison a cache. They are currently regular folders that are read/writable to all users, at a global location on the filesystem.

The docs for mounts/caches in generic-worker can curently be found here:
  * https://docs.taskcluster.net/reference/workers/generic-worker/payload

A real-world example of a generic-worker scope-protected, non-preloaded cache is here:
  * https://github.com/taskcluster/taskcluster-worker/blob/319fa133caa43f78368f77dff02a6e8da1152faa/.taskcluster.yml#L106-L107

The above example is from the taskcluster-worker CI, which currently runs under generic-worker (i.e. this is a generic-worker feature not a taskcluster-worker feature).

It looks like e.g. vcs caching is handled in gecko task generation here, but currently only supports docker worker:
  * https://hg.mozilla.org/mozilla-central/file/9577ddeaafd8/taskcluster/taskgraph/transforms/job/common.py#l103

There is probably also something similar for object caches.

If we can get the gecko task generation to take advantage of generic worker caches for object caches and vcs caches, we should gain more task transparency (more defined in task, less defined in the AMI) which also improves portability, and reusability of worker types, as well as increasing security by having caches protected by scopes.
Is the action here to implement scope protection, or to use the caches in the in-tree taskgraph generation even though they are not scope-protected?
Flags: needinfo?(pmoore)
(Assignee)

Comment 2

8 months ago
(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> Is the action here to implement scope protection, or to use the caches in
> the in-tree taskgraph generation even though they are not scope-protected?

The in-tree task generation of gecko should generate generic-worker build tasks that mount writable vcs cache and object directory caches via the "mounts" pragma of the task payload. Use of writable cache <cache-name> necessarily requires scope generic-worker:cache:<cache-name>, which provides fine-grained access control.
Flags: needinfo?(pmoore)
Ah, I misunderstood comment 0 to say that generic-worker didn't support scope-protected caches.  It does, but we're not using them in-tree.  Thanks!
Mentor: dustin@mozilla.com
Component: General Automation → Task Configuration
Keywords: good-first-bug
Product: Release Engineering → Taskcluster
QA Contact: catlee
(Assignee)

Updated

8 months ago
Assignee: nobody → pmoore
(Assignee)

Comment 4

8 months ago
I'm happy to take a stab at this, with my new knoweldge of the in-tree task generation process from bug 1349980.

I'll be on PTO for a week after today, so won't get done until later in the month, as I need to sort out rebooting on generic worker on OS X in scl3 first before I leave.
You need to log in before you can comment on or make changes to this bug.