1285506 - Deploy AMIs for win2012 taskcluster worker type in us-west-1 and eu-central-1

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Pete Moore [:pmoore][:pete]]

Before we can do this, we need to make sure s3 cache buckets/clone bundles exist in those regions...

Comment 1

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for OpenCloudConfig

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8773252 [details] [review]
Github Pull Request for OpenCloudConfig

merged: https://github.com/mozilla-releng/OpenCloudConfig/commit/00c47dfdc3a8cb9709c9b4bb24417024fc4dc5a0

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

policy updated to include Frankfurt (https://console.aws.amazon.com/iam/home?region=us-west-2#policies/arn:aws:iam::692406183521:policy/taskcluster-level-1-sccache):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1467642244000",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::taskcluster-level-1-sccache-eu-central-1/*",
                "arn:aws:s3:::taskcluster-level-1-sccache-us-east-1/*",
                "arn:aws:s3:::taskcluster-level-1-sccache-us-west-1/*",
                "arn:aws:s3:::taskcluster-level-1-sccache-us-west-2/*"
            ]
        },
        {
            "Sid": "Stmt1467642244001",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetBucketTagging",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::taskcluster-level-1-sccache-eu-central-1",
                "arn:aws:s3:::taskcluster-level-1-sccache-us-east-1",
                "arn:aws:s3:::taskcluster-level-1-sccache-us-west-1",
                "arn:aws:s3:::taskcluster-level-1-sccache-us-west-2"
            ]
        }
    ]
}

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: Bug 1285506 - include eu-central-1 in tc sccache bucket regions;

Review commit: https://reviewboard.mozilla.org/r/66890/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/66890/

Comment 5

[Rob Thijssen [:grenade (EET/UTC+0300)]]

jhford: do we need to modify cloud-mirror to support building in eu-central-1

Comment 6

[Mike Hommey [:glandium]]

Comment on attachment 8774428 [details]
Bug 1285506 - include eu-central-1 in tc sccache bucket regions;

https://reviewboard.mozilla.org/r/66890/#review63932

Comment 7

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8774428 [details]
Bug 1285506 - include eu-central-1 in tc sccache bucket regions;

https://reviewboard.mozilla.org/r/66890/#review63942

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/784f2a4f0d22
include eu-central-1 in tc sccache bucket regions; r=glandium

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

Rob, we currently have eu-central-1 enabled in cloud-mirror.

Here's an example image that's cached there.  Let me double check that the queue will rewrite to that location as well.

https://cloud-mirror.taskcluster.net/v1/redirect/s3/eu-central-1/https%3A%2F%2Fwww.mozilla.org%2Fmedia%2Fimg%2Fhome%2Fvoices%2Fmosaic-background.565019b6e6bc.jpg

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Rob, we did not have url rewriting on the cache side enabled for cloud-mirror.  I've written a patch for it and asked for review.  cloud-mirror will not be effective until this patch lands.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: queue patch

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/784f2a4f0d22

Comment 13

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/taskcluster-queue

https://github.com/taskcluster/taskcluster-queue/commit/7f3205db23f5f37b3718b2ca03a806f84808bb1d
bug 1285506 -- let queue rewrite urls for eu-central-1

https://github.com/taskcluster/taskcluster-queue/commit/98b90ddfdedb6c96cd8b6d197149cf7576ffdef0
Merge pull request #106 from taskcluster/enable-eu-central-1

bug 1285506 -- let queue rewrite urls for eu-central-1

Comment 14

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8774718 [details] [review]
queue patch

In the process of landing...

Comment 15

[Rob Thijssen [:grenade (EET/UTC+0300)]]

gps: do you have any concerns or thoughts regarding hg clonebundles or cloning from builders in eu-central-1? is it needed|difficult|possible to set up bundle caches in eu-central-1 to support builds there?

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/53996cc0b213
hgserver: upload bundles to eu-central-1

Comment 17

[Gregory Szorc [:gps]]

Thank you for notifying me about spinning up automation in a new region so we can configure the Mercurial bundles in that region. If we fail to configure the bundles, it isn't the end of the world: bundles will be downloaded from the CDN instead of S3 and we'll get hit with a CDN bill. But I strongly prefer having the bundles in place from day 0.

I've triggered manual generation of the bundles, so they should start appearing in eu-central-1 within an hour or so.

Comment 18

[Pete Moore [:pmoore][:pete]]

We're actively using eu-central-1 and us-west-1 so closing this now.

Comment 19

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.