966152 - Create two S3 buckets and make them available from build slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Hommey [:glandium]]

Now that the test phase is over, to deploy shared cache on S3, we need to setup two buckets, one in each region (use1, usw2). Note this means that if we ever deploy build infrastructure on another region, that will need to be part of the deal.

On those buckets, there needs to be a bucket policy to make objects public readable (makes download faster than requiring authentication). The bucket policy i've been using during testing was the following (replace {bucket_name} with the relevant name):
{
	"Version": "2008-10-17",
	"Statement": [
		{
			"Sid": "AllowPublicRead",
			"Effect": "Allow",
			"Principal": {
				"AWS": "*"
			},
			"Action": "s3:GetObject",
			"Resource": "arn:aws:s3:::{bucket_name}/*"
		}
	]
}

We should also set an expiration on all objects in the bucket. I suggest 15 days from creation start.

We also need an IAM policy for the build slaves:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPut",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::{bucket_name}/*"
      ]
    }
  ]
}

Comment 1

[Mike Hommey [:glandium]]

According to the last comments in bug 966301, try spot instances have an IAM role now. Can we go ahead with the buckets so that we can at least test on try?

Comment 2

[Rail Aliiev [:rail]]

I suggest to create separate buckets for production and try pools. We use separate pools of machines for try and other branches to address security concerns. The same may apply to S3 cache.

Comment 3

[Rail Aliiev [:rail]]

I created 4 buckets:

Production:
mozilla-releng-s3-cache-us-east-1-prod
mozilla-releng-s3-cache-us-west-2-prod

Try:
mozilla-releng-s3-cache-us-east-1-try
mozilla-releng-s3-cache-us-west-2-try

All 4 have a life cycle rule deleting objects older than 15 days.

I attached a read-only policy to these buckets.

In IAM, I created the following role policies:

try-linux64: can read/write to both *try* buckets (to avoid region specific role policies)
dev-linux64: can read/write to both *try* buckets
bld-linux64: can read/write to both *prod* buckets

ATM, the roles are available only for spot instances (on-demand instances still to be recreated).

Comment 4

[Mike Hommey [:glandium]]

I'm getting 403 errors on HTTP GET on try. I guess this is a bucket policy problem.

Comment 5

[Mike Hommey [:glandium]]

This was e.g. from try-linux64-spot-355 to mozilla-releng-s3-cache-us-west-2-try

Comment 6

[Mike Hommey [:glandium]]

(Note that PUT did work properly afaict)

Comment 7

[Rail Aliiev [:rail]]

Hmm, this is weird...

The S3 policy looks like this:

{
	"Version": "2008-10-17",
	"Statement": [
		{
			"Sid": "AllowPublicRead",
			"Effect": "Allow",
			"Principal": {
				"AWS": "*"
			},
			"Action": "s3:GetObject",
			"Resource": "arn:aws:s3:::mozilla-releng-s3-cache-us-east-1-try/*"
		}
	]
}

I can see a lot of objects created under this bucket, which means that PutObject works as expected.

Investigating...

Comment 8

[Rail Aliiev [:rail]]

The corresponding role policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPut",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::mozilla-releng-s3-cache-us-west-2-try/*"
      ]
    }
  ]
}

BTW, I think we will also need to add permission to get/put objects for inhouse slaves as well (by IP?).

Comment 9

[Mike Hommey [:glandium]]

(In reply to Rail Aliiev [:rail] from comment #8)
> BTW, I think we will also need to add permission to get/put objects for
> inhouse slaves as well (by IP?).

We're not going touse S3 for inhouse slaves. Too much latency.

Comment 10

[Mike Hommey [:glandium]]

So it turns out S3 returns 403 for missing objects. I had broken exception handling in my test and instead of gracefully falling back to compiling when there was a cache miss, it was throwing. Fixing that made it work properly.