867792 - popcorn.webmadecontent.org and other S3 roots shouldn't serve the bucket list

Status: RESOLVED FIXED

(Webmaker Graveyard :: DevOps, defect)

Description

[David Humphrey (:humph)]

If you go to http://popcorn.webmadecontent.org/ you'll get S3 serving you an XML doc with the contents of the bucket.  That's not ideal, to say the least.

We should get some kind of redirect in place that pushes any access of / in our S3 buckets to webmaker.org.

Also CC'ing cmac so he's aware for Badges/CSOL buckets.

Comment 1

[JP Schneider :jp]

This was the policy in place:

{
	"Version": "2008-10-17",
	"Statement": [
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": {
				"AWS": "*"
			},
			"Action": [
				"s3:ListBucket",
				"s3:GetObject"
			],
			"Resource": [
				"arn:aws:s3:::popcorn.webmadecontent.org",
				"arn:aws:s3:::popcorn.webmadecontent.org/*"
			]
		}
	]
}

I have removed the listing permissions, and am verifying that everything continues to work.

I am working on a conditional routing rule to redirect the root, will update shortly.

Comment 2

[JP Schneider :jp]

FWIW, we actually chose this explicitly before, so we would see 404's instead of 403's for files that did not exist.

--
From AWS

Hello JP,

I'm following up your support request regarding issues with your S3 bucket popcorn.webmadecontent.org

Currently your bucket policy is configured as follows:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::popcorn.webmadecontent.org/*"
        }
    ]
}

I tested the policy on a replica scenario and it works as expected: if the object is in the bucket, it lets public access to it.

However, when a requested object is not present, it sends the "403" error code ("Access Denied").

In the excerpt of log you sent us, I see some of the objects giving the 403 error are (listing a few as an example):

/user_images/550ba37d-9dac-414f-b563-de404ee1c69f 
/user_images/c4f9ef1e-fb91-444b-b7dc-528cce3b7a01
/user_images/4015556f-fa33-4a6f-85c0-4f731817c741
/user_images/58bff144-41c6-4097-ae52-bc160cd1e171

The error means that those objects are not present in the bucket. Can you confirm that?

Let me explain why you are getting 403 ("Access Denied") error instead of the more intuitive 404 ("Not Found"): the bucket and the objects inside it are not fully public, the policy you are currently using are just allowing everyone to execute the "GetObject" action against the objects.

The GetObject action can be used only to access to an object: if this is not possible, whether because of a different policy for the specific object or just because it does not exist, it just says that the object cannot be accessed, thus the 403 error.

If you want S3 to explicitly says if an object exists or not, you have to enable another action, the ListBucket.

Please note that this will also allow let the bucket to be listed for all the present objects, which probably is not what you want.

In any case the policy to enable that is the following:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::popcorn.webmadecontent.org/*",
                "arn:aws:s3:::popcorn.webmadecontent.org"
            ]
        }
    ]
}

Please note the slight difference between the two resources: the one with /* is referred to the objects inside the bucket (as the GetObject is an action on the objects), while the other is the bucket itself (ListBucket being a bucket related action).

You can read a deeper explanation of this on our documentation for S3 Policies:
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingResOpsConditions.html

Please let me know if you need to discuss any further this issue.

Best regards,

Lorenzo L.

Comment 3

[Mike Beltzner [:beltzner, not reading bugmail]]

Still needed?

Comment 4

[JP Schneider :jp]

Closing this out.  I've heard no further feedback about these urls serving dir