1601125 - Taskcluster 24.1.1 deploy tracking

Status: RESOLVED FIXED

(Cloud Services :: Operations: Taskcluster, task)

Description

[edunham]

Deploying to stage, then community with TC team's ok, then to firefoxci. Before deploy to firefoxci, I'll give a day of notice to #firefoxci, mentioning the release duty folks listed in https://wiki.mozilla.org/Release_Management/Release_owners.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Due to bug 1601149 we have released 24.0.1, and recommend upgrading directly to that version, skipping 24.0.0.

Comment 2

[edunham]

Updated topic to reflect v24.0.1.

Stage is currently running v24.0.1. Please let me know what you'd like the buckets parameter of aws_credentials_allowed_buckets to look like for both communitytc and firefoxcitc, and I'll update their configs when we're ready to roll them out. I'd like to get 24.0.1 onto community today and give the #firefox-ci channel the heads up about it, then roll it out to firefoxci Monday if no problems are found with 24.0.1 on stage.

Notified release duty about this in https://mozilla.slack.com/archives/C7JJBN3MM/p1575576188431200

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

For community-tc, there's no need for any aws_credentials_allowed_buckets (omit it or set it to {}).

For firefox-tc-ci, there are some buckets. The current policies are using the IAM user auth.taskcluster.net, which is a little awkward! I made a new user in the mozilla-taskcluster account and applied the exact same (sccache) policies.

wormhole receive 3-chicago-music

# this is the firefox-ci-tc-sccache AWS user in the mozilla-taskcluster account
- accessKeyId: ***
  secretAccessKey: ***
  buckets:
  - taskcluster-level-1-sccache-eu-central-1
  - taskcluster-level-1-sccache-us-east-1
  - taskcluster-level-1-sccache-us-west-1
  - taskcluster-level-1-sccache-us-west-2
  - taskcluster-level-2-sccache-eu-central-1
  - taskcluster-level-2-sccache-us-east-1
  - taskcluster-level-2-sccache-us-west-1
  - taskcluster-level-2-sccache-us-west-2
  - taskcluster-level-3-sccache-eu-central-1
  - taskcluster-level-3-sccache-us-east-1
  - taskcluster-level-3-sccache-us-west-1
  - taskcluster-level-3-sccache-us-west-2
  - comm-central-level-1-sccache-us-east-1
  - comm-central-level-1-sccache-us-east-2
  - comm-central-level-1-sccache-us-west-1
  - comm-central-level-1-sccache-us-west-2
  - comm-central-level-1-sccache-eu-central-1
  - comm-central-level-2-sccache-us-east-1
  - comm-central-level-2-sccache-us-east-2
  - comm-central-level-2-sccache-us-west-1
  - comm-central-level-2-sccache-us-west-2
  - comm-central-level-2-sccache-eu-central-1
  - comm-central-level-3-sccache-us-east-1
  - comm-central-level-3-sccache-us-east-2
  - comm-central-level-3-sccache-us-west-1
  - comm-central-level-3-sccache-us-west-2
  - comm-central-level-3-sccache-eu-central-1

Comment 4

[edunham]

TC 24.0.1 is live on Communitytc and Firefoxcitc now.

Comment 5

[edunham]

Reverted fxci to v23 due to nightly breakage.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks for dealing with that! Sorry I had to leave at a "normal" hour yesterday and was afk :(

So if I read the scrollback correctly:

• there was a typo (allowdBuckets) (bug 1602642)
• there were some buckets missed in the config, at least tc-gp-private-1d-us-east-1. Were there others?

I'm baking a 24.0.2 based on 24.0.1 right now, with the typo fixed. Let's get that deployed to staging ASAP.

The missed buckets will need to be updated both in the IAM policy and in the TC deployment config. Were both of those done during the debugging? Happily if we find anything else we missed, it's easy to update without rollback.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

OK, 24.0.2 is ready. Let's get that on staging, and for the moment copy/pasta the allowed buckets config from production to staging. We can make some API calls to verify that it works for the known buckets, which should give some more confidence before rolling to firefox-ci. It's my understanding that we do not want to share these buckets between staging and production, so this will just be a short-term thing. But, hopefully we can get 24.0.2 out to firefox-ci in the mid-day US/Pacific timeframe today.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I checked over the IAM policies and they look good. I renamed the tc-gp-private-1d-related policy and also adjusted it to match the others (with separate statements for the bucket and the enclosed objects). Should be good to go!

Comment 9

[edunham]

24.0.2 is now out. Config changes to resume 24.x compatibility in firefoxci are pushed to sops, so a deploy will be all it takes to propagate it.

Stage should have auto-updated from 24.x to 24.0.2.

Per Slack conversation, 24.0.2 is a one-line patch that's irrelevant to community, so we'll skip it there.

Comment 10

[edunham]

And now we're 24.1.1. It's on stage and community, firefoxci later today.

Comment 11

[edunham]

Should have closed this awhile ago