Closed Bug 1423801 Opened 6 years ago Closed 6 years ago

Create sccache buckets for comm-central derived builds.

Categories

(Taskcluster :: Operations and Service Requests, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: tomprince, Assigned: tomprince)

References

Details

Attachments

(2 files, 1 obsolete file)

I'd like to enable sccache for thunderbird builds, and will need level-1 and level-3 S3 buckets for that.
Flags: needinfo?(jopsen)
Sorry, for being slow to get on with this...

I've started to create some bucket, managed with terraform... Anyways, I filed with AWS to get a bucket increase as we need more buckets :)
Created buckets:

  comm-central-level-1-sccache-eu-central-1
  comm-central-level-2-sccache-eu-central-1
  comm-central-level-3-sccache-eu-central-1
  comm-central-level-1-sccache-us-east-2
  comm-central-level-2-sccache-us-east-2
  comm-central-level-3-sccache-us-east-2
  comm-central-level-1-sccache-us-west-1
  comm-central-level-2-sccache-us-west-1
  comm-central-level-3-sccache-us-west-1
  comm-central-level-1-sccache-us-west-2
  comm-central-level-2-sccache-us-west-2
  comm-central-level-3-sccache-us-west-2

Attached to taskcluster-auth, so credentials can be created using:
  https://docs.taskcluster.net/reference/platform/taskcluster-auth/references/api#awsS3Credentials

Also attached to the same IAM roles as gecko level-1 and level-3.
But I'm guessing comm-central won't be running on gecko level-3 builders, so that might not be important.

@dustin, we should grant the scope:
    auth:aws-s3:read-write:<bucket>/*
To the right appropriate comm-central roles... I'm guessing you have try-like thing that needs the level-1 scopes
and some repo role that needs level-3. These seems to be tc-admin managed, what would you suggest?
Flags: needinfo?(jopsen) → needinfo?(dustin)
It looks like the way these are managed for gecko is that the auth scopes are defined in roles `project:taskcluster:level-L-sccache-buckets`
  https://tools.taskcluster.net/auth/roles/project%3Ataskcluster%3Alevel-1-sccache-buckets
That role is then assumed in `moz-tree:level:N`
  https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A1
and that role is, in turn, assumed in `repo:hg.mozilla.org/try:*`
  https://tools.taskcluster.net/auth/roles/repo%3Ahg.mozilla.org%2Ftry%3A*

There is a try-comm-central repo:
  https://tools.taskcluster.net/auth/roles/repo%3Ahg.mozilla.org%2Ftry-comm-central%3A*
and that, too, assumes moz-tree:level:1

If we do not want to blaze a new trail here, we can just assign the scopes to `project:taskcluster:level-L-sccache-buckets`.

If, as I suspect, the idea is to separate sccache by trust domain, then maybe we need to split the `moz-tree:level:L` roles by trust domain, too.  We can split them but leave their scopes identical except for sccache at first, then continue to chip away a the shared resources until they are completely separated.

So: which should we do?
Flags: needinfo?(dustin)
I think we do eventually want to split the roles. As a first step, maybe we want to do the following:

- Create moz-tree:level:<N>:<trust-domain> and add that to the tc-admin settings.
- Add project:taskcluster:comm:level-<n>-sccache-buckets to moz-tree:level:<N>:comm (under project:taskcluster since they are in the TC AWS account)
- Do the same for trust-domain = gecko. (this could also wait for the new year)

In the new year, clean things up by removing the role project:taskcluster:level-<n>-sccache-bucket in favor of the new roles.
OK, agreed.  I set up `moz-tree:level:N:*` with the same scopes as `moz-tree:level:N` used to have, then added `moz-tree:level:N:gecko` and `moz-tree:level:N:comm`.  The latter two implement the "cascade" where L2 implies L1 and L3 implies L2 and L1.

  https://tools.taskcluster.net/auth/roles/moz-tree:level:1
  https://tools.taskcluster.net/auth/roles/moz-tree:level:1:*
  https://tools.taskcluster.net/auth/roles/moz-tree:level:1:gecko
  https://tools.taskcluster.net/auth/roles/moz-tree:level:1:comm
  https://tools.taskcluster.net/auth/roles/moz-tree:level:2
  https://tools.taskcluster.net/auth/roles/moz-tree:level:2:*
  https://tools.taskcluster.net/auth/roles/moz-tree:level:2:gecko
  https://tools.taskcluster.net/auth/roles/moz-tree:level:2:comm
  https://tools.taskcluster.net/auth/roles/moz-tree:level:3
  https://tools.taskcluster.net/auth/roles/moz-tree:level:3:*
  https://tools.taskcluster.net/auth/roles/moz-tree:level:3:gecko
  https://tools.taskcluster.net/auth/roles/moz-tree:level:3:comm

Then I moved the gecko sccache roles:

  https://tools.taskcluster.net/auth/roles/project%3Ataskcluster%3Agecko%3Alevel-1-sccache-buckets
  https://tools.taskcluster.net/auth/roles/project%3Ataskcluster%3Agecko%3Alevel-2-sccache-buckets
  https://tools.taskcluster.net/auth/roles/project%3Ataskcluster%3Agecko%3Alevel-3-sccache-buckets

and created new comm sccache roles

  https://tools.taskcluster.net/auth/roles/project%3Ataskcluster%3Acomm%3Alevel-1-sccache-buckets
  https://tools.taskcluster.net/auth/roles/project%3Ataskcluster%3Acomm%3Alevel-2-sccache-buckets
  https://tools.taskcluster.net/auth/roles/project%3Ataskcluster%3Acomm%3Alevel-3-sccache-buckets

and finally edited the moz-tree roles to include the appropriate sccache roles.
Oops, "assume:project:taskcluster:level-N-sccache-buckets" appears directly in the scopes used in-tree.  I re-added those roles as aliases to the new :gecko: roles.  Tom, I'll leave it to you to make a patch to use the new role names (since that will need to include the trust domain)
I had to add those scopes to moz-tree:level:N:gecko too.  That will need to be reverted eventually.
Keywords: leave-open
Comment on attachment 8938529 [details]
Bug 1423801: Include trust-domain in the scopes to use for sccache;

https://reviewboard.mozilla.org/r/209192/#review215012
Attachment #8938529 - Flags: review?(dustin) → review+
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/a931893981e5
Include trust-domain in the scopes to use for sccache; r=dustin
I've got this partly working in a branch. There are a couple of issues:

- sccache doesn't use keys when trying to read from the cache, so the buckets need to be world-readable.
- I think the list of regions doesn't match the regions that workers are in. I know at least us-east-1 has workers but no buckets.
@tomprince, okay.. I'll:
 - add us-east-1, and,
 - make them public readable
Attachment #8938465 - Attachment is obsolete: true
Comment on attachment 8939701 [details]
Bug 1423801: Use comm-central sccache buckets;

https://reviewboard.mozilla.org/r/210004/#review215832
Attachment #8939701 - Flags: review?(philipp) → review+
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/comm-central/rev/678d7a3b822a
Use comm-central sccache buckets; r?Fallen
Blocks: 1429836
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 1434128
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Assignee: nobody → mozilla
Component: Service Request → Operations and Service Requests
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: