1181892 - enable anonymous access to s3 release buckets to list and retrieve objects

Status: VERIFIED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[[e-mail] (:bhearsum)]

Part of our release automation collects files out of candidates directories (like http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0b2-candidates/) to parse and collate their contents. I've written a script that does this directly on our new S3 buckets, but it requires the ability to list the objects in the bucket and retrieve them individually. There's nothing private in these buckets as far as I know, so this should be safe to do...

I poked about this over e-mail originally, I'm filing a bug to track this better.

Comment 1

[Jeremy Orem [:oremj]]

This has been added to the bucket creation scripts and done in stage.

Comment 2

[[e-mail] (:bhearsum)]

Looks like anonymous listing has been enabled, but not anonymous retrieval of items:
13:02:09    DEBUG - Downloading pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - path=/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - auth_path=/net-mozaws-stage-delivery-firefox/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - Method: HEAD
13:02:09    DEBUG - Path: /pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - Data: 
13:02:09    DEBUG - Headers: {}
13:02:09    DEBUG - Host: net-mozaws-stage-delivery-firefox.s3.amazonaws.com
13:02:09    DEBUG - Port: 443
13:02:09    FATAL - Uncaught exception: Traceback (most recent call last):
13:02:09    DEBUG - Params: {}
13:02:09    DEBUG - Token: None
13:02:09    DEBUG - Final headers: {'Content-Length': '0', 'User-Agent': 'Boto/2.38.0 Python/2.7.9 Linux/3.19.0-25-generic'}
13:02:09    FATAL -   File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1693, in run
13:02:09    FATAL -     self.run_action(action)
13:02:09    FATAL -   File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1635, in run_action
13:02:09    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
13:02:09    FATAL -   File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1576, in _possibly_run_method
13:02:09    FATAL -     return getattr(self, method_name)()
13:02:09    FATAL -   File "generate-checksums.py", line 175, in collect_individual_checksums
13:02:09    FATAL -     pool.map(worker, find_checksums_files())
13:02:09    FATAL -   File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
13:02:09    FATAL -     return self.map_async(func, iterable, chunksize).get()
13:02:09    FATAL -   File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
13:02:09    FATAL -     raise self._value
13:02:09    FATAL - S3ResponseError: S3ResponseError: 403 Forbidden
13:02:09    FATAL - <?xml version="1.0" encoding="UTF-8"?>
13:02:09    FATAL - <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>3D9A90B1D8E787CA</RequestId><HostId>NHtEMMoxm5iTSrfAUeDviI5bc7fhyXm7eH3Dpkvmdx077XOmgBxnX/i0tMuJe4BG</HostId></Error>

Comment 3

[Jeremy Orem [:oremj]]

Hmm, it's working for me
> curl -I http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
> HTTP/1.1 200 OK
> x-amz-id-2: kZB+uU70UAoKYlf4O27Em5Ll4dye7hwgkPAh6hIKr6TZbUmtZa5y74e4yV4pHkuZ4WG8dFxd+f4= 
> x-amz-request-id: F57EA593EEDF6237
> Date: Fri, 07 Aug 2015 20:20:09 GMT
> x-amz-version-id: jorWBVA4iwZq4LVJnAl_uMPXNCp5T7gG
> Last-Modified: Fri, 07 Aug 2015 16:07:22 GMT
> ETag: "88ed2808e06e055c6f5acd795e65d8ff"
> Accept-Ranges: bytes
> Content-Type: binary/octet-stream
> Content-Length: 4068
> Server: AmazonS3

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Jeremy Orem [:oremj] from comment #3)
> Hmm, it's working for me
> > curl -I http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
> > HTTP/1.1 200 OK
> > x-amz-id-2: kZB+uU70UAoKYlf4O27Em5Ll4dye7hwgkPAh6hIKr6TZbUmtZa5y74e4yV4pHkuZ4WG8dFxd+f4= 
> > x-amz-request-id: F57EA593EEDF6237
> > Date: Fri, 07 Aug 2015 20:20:09 GMT
> > x-amz-version-id: jorWBVA4iwZq4LVJnAl_uMPXNCp5T7gG
> > Last-Modified: Fri, 07 Aug 2015 16:07:22 GMT
> > ETag: "88ed2808e06e055c6f5acd795e65d8ff"
> > Accept-Ranges: bytes
> > Content-Type: binary/octet-stream
> > Content-Length: 4068
> > Server: AmazonS3

Looks like the difference is that Boto appends versionId as a query arg:
DEBUG:boto:Path: /pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY


If I add that to a curl request, I get a 403:
curl "http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY"
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B73686F8406D0C90</RequestId><HostId>thMxCXxo7vcA99TPNBvzR10HSNTGy4B3K6U6NltmZF+ppFz6lOIC083zq4tj35syBCTBZZ5CloY=</HostId></Error>%

Comment 5

[Jeremy Orem [:oremj]]

I added an allow on GetObjectVersion

curl -I "http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY"
HTTP/1.1 200 OK
x-amz-id-2: kiswmXN8FRncReqvfjVW9PY0/uYOSMEOEbX3ST/dYSKAYtXb/wUXb5PXWixNxDG1
x-amz-request-id: 0448593F0C55C9F8
Date: Mon, 10 Aug 2015 15:54:38 GMT
x-amz-version-id: dH1FAUCOdaUW1.CwfbQUpB12W52idKbY
Last-Modified: Fri, 07 Aug 2015 16:14:23 GMT
ETag: "f124f465aefc0a8cc68ec827ca7475e5"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 4122
Server: AmazonS3

Comment 6

[[e-mail] (:bhearsum)]

Looks good now, thanks Jeremy!