enable anonymous access to s3 release buckets to list and retrieve objects

VERIFIED FIXED

Status

VERIFIED FIXED
3 years ago
3 years ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
Part of our release automation collects files out of candidates directories (like http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0b2-candidates/) to parse and collate their contents. I've written a script that does this directly on our new S3 buckets, but it requires the ability to list the objects in the bucket and retrieve them individually. There's nothing private in these buckets as far as I know, so this should be safe to do...

I poked about this over e-mail originally, I'm filing a bug to track this better.

Comment 1

3 years ago
This has been added to the bucket creation scripts and done in stage.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 2

3 years ago
Looks like anonymous listing has been enabled, but not anonymous retrieval of items:
13:02:09    DEBUG - Downloading pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - path=/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - auth_path=/net-mozaws-stage-delivery-firefox/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - Method: HEAD
13:02:09    DEBUG - Path: /pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
13:02:09    DEBUG - Data: 
13:02:09    DEBUG - Headers: {}
13:02:09    DEBUG - Host: net-mozaws-stage-delivery-firefox.s3.amazonaws.com
13:02:09    DEBUG - Port: 443
13:02:09    FATAL - Uncaught exception: Traceback (most recent call last):
13:02:09    DEBUG - Params: {}
13:02:09    DEBUG - Token: None
13:02:09    DEBUG - Final headers: {'Content-Length': '0', 'User-Agent': 'Boto/2.38.0 Python/2.7.9 Linux/3.19.0-25-generic'}
13:02:09    FATAL -   File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1693, in run
13:02:09    FATAL -     self.run_action(action)
13:02:09    FATAL -   File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1635, in run_action
13:02:09    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
13:02:09    FATAL -   File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1576, in _possibly_run_method
13:02:09    FATAL -     return getattr(self, method_name)()
13:02:09    FATAL -   File "generate-checksums.py", line 175, in collect_individual_checksums
13:02:09    FATAL -     pool.map(worker, find_checksums_files())
13:02:09    FATAL -   File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
13:02:09    FATAL -     return self.map_async(func, iterable, chunksize).get()
13:02:09    FATAL -   File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
13:02:09    FATAL -     raise self._value
13:02:09    FATAL - S3ResponseError: S3ResponseError: 403 Forbidden
13:02:09    FATAL - <?xml version="1.0" encoding="UTF-8"?>
13:02:09    FATAL - <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>3D9A90B1D8E787CA</RequestId><HostId>NHtEMMoxm5iTSrfAUeDviI5bc7fhyXm7eH3Dpkvmdx077XOmgBxnX/i0tMuJe4BG</HostId></Error>
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 3

3 years ago
Hmm, it's working for me
> curl -I http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
> HTTP/1.1 200 OK
> x-amz-id-2: kZB+uU70UAoKYlf4O27Em5Ll4dye7hwgkPAh6hIKr6TZbUmtZa5y74e4yV4pHkuZ4WG8dFxd+f4= 
> x-amz-request-id: F57EA593EEDF6237
> Date: Fri, 07 Aug 2015 20:20:09 GMT
> x-amz-version-id: jorWBVA4iwZq4LVJnAl_uMPXNCp5T7gG
> Last-Modified: Fri, 07 Aug 2015 16:07:22 GMT
> ETag: "88ed2808e06e055c6f5acd795e65d8ff"
> Accept-Ranges: bytes
> Content-Type: binary/octet-stream
> Content-Length: 4068
> Server: AmazonS3
(Reporter)

Comment 4

3 years ago
(In reply to Jeremy Orem [:oremj] from comment #3)
> Hmm, it's working for me
> > curl -I http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
> > HTTP/1.1 200 OK
> > x-amz-id-2: kZB+uU70UAoKYlf4O27Em5Ll4dye7hwgkPAh6hIKr6TZbUmtZa5y74e4yV4pHkuZ4WG8dFxd+f4= 
> > x-amz-request-id: F57EA593EEDF6237
> > Date: Fri, 07 Aug 2015 20:20:09 GMT
> > x-amz-version-id: jorWBVA4iwZq4LVJnAl_uMPXNCp5T7gG
> > Last-Modified: Fri, 07 Aug 2015 16:07:22 GMT
> > ETag: "88ed2808e06e055c6f5acd795e65d8ff"
> > Accept-Ranges: bytes
> > Content-Type: binary/octet-stream
> > Content-Length: 4068
> > Server: AmazonS3

Looks like the difference is that Boto appends versionId as a query arg:
DEBUG:boto:Path: /pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY


If I add that to a curl request, I get a 403:
curl "http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY"
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B73686F8406D0C90</RequestId><HostId>thMxCXxo7vcA99TPNBvzR10HSNTGy4B3K6U6NltmZF+ppFz6lOIC083zq4tj35syBCTBZZ5CloY=</HostId></Error>%

Comment 5

3 years ago
I added an allow on GetObjectVersion

curl -I "http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY"
HTTP/1.1 200 OK
x-amz-id-2: kiswmXN8FRncReqvfjVW9PY0/uYOSMEOEbX3ST/dYSKAYtXb/wUXb5PXWixNxDG1
x-amz-request-id: 0448593F0C55C9F8
Date: Mon, 10 Aug 2015 15:54:38 GMT
x-amz-version-id: dH1FAUCOdaUW1.CwfbQUpB12W52idKbY
Last-Modified: Fri, 07 Aug 2015 16:14:23 GMT
ETag: "f124f465aefc0a8cc68ec827ca7475e5"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 4122
Server: AmazonS3
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 6

3 years ago
Looks good now, thanks Jeremy!
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.