Closed
Bug 1181892
Opened 9 years ago
Closed 9 years ago
enable anonymous access to s3 release buckets to list and retrieve objects
Categories
(Cloud Services :: Operations: Miscellaneous, task)
Cloud Services
Operations: Miscellaneous
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: bhearsum, Unassigned)
References
Details
Part of our release automation collects files out of candidates directories (like http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0b2-candidates/) to parse and collate their contents. I've written a script that does this directly on our new S3 buckets, but it requires the ability to list the objects in the bucket and retrieve them individually. There's nothing private in these buckets as far as I know, so this should be safe to do... I poked about this over e-mail originally, I'm filing a bug to track this better.
Comment 1•9 years ago
|
||
This has been added to the bucket creation scripts and done in stage.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 2•9 years ago
|
||
Looks like anonymous listing has been enabled, but not anonymous retrieval of items: 13:02:09 DEBUG - Downloading pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums 13:02:09 DEBUG - path=/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums 13:02:09 DEBUG - auth_path=/net-mozaws-stage-delivery-firefox/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums 13:02:09 DEBUG - Method: HEAD 13:02:09 DEBUG - Path: /pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums 13:02:09 DEBUG - Data: 13:02:09 DEBUG - Headers: {} 13:02:09 DEBUG - Host: net-mozaws-stage-delivery-firefox.s3.amazonaws.com 13:02:09 DEBUG - Port: 443 13:02:09 FATAL - Uncaught exception: Traceback (most recent call last): 13:02:09 DEBUG - Params: {} 13:02:09 DEBUG - Token: None 13:02:09 DEBUG - Final headers: {'Content-Length': '0', 'User-Agent': 'Boto/2.38.0 Python/2.7.9 Linux/3.19.0-25-generic'} 13:02:09 FATAL - File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1693, in run 13:02:09 FATAL - self.run_action(action) 13:02:09 FATAL - File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1635, in run_action 13:02:09 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 13:02:09 FATAL - File "/home/bhearsum/tmp/checksums/mozharness/mozharness/base/script.py", line 1576, in _possibly_run_method 13:02:09 FATAL - return getattr(self, method_name)() 13:02:09 FATAL - File "generate-checksums.py", line 175, in collect_individual_checksums 13:02:09 FATAL - pool.map(worker, find_checksums_files()) 13:02:09 FATAL - File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map 13:02:09 FATAL - return self.map_async(func, iterable, chunksize).get() 13:02:09 FATAL - File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get 13:02:09 FATAL - raise self._value 13:02:09 FATAL - S3ResponseError: S3ResponseError: 403 Forbidden 13:02:09 FATAL - <?xml version="1.0" encoding="UTF-8"?> 13:02:09 FATAL - <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>3D9A90B1D8E787CA</RequestId><HostId>NHtEMMoxm5iTSrfAUeDviI5bc7fhyXm7eH3Dpkvmdx077XOmgBxnX/i0tMuJe4BG</HostId></Error>
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 3•9 years ago
|
||
Hmm, it's working for me
> curl -I http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums
> HTTP/1.1 200 OK
> x-amz-id-2: kZB+uU70UAoKYlf4O27Em5Ll4dye7hwgkPAh6hIKr6TZbUmtZa5y74e4yV4pHkuZ4WG8dFxd+f4=
> x-amz-request-id: F57EA593EEDF6237
> Date: Fri, 07 Aug 2015 20:20:09 GMT
> x-amz-version-id: jorWBVA4iwZq4LVJnAl_uMPXNCp5T7gG
> Last-Modified: Fri, 07 Aug 2015 16:07:22 GMT
> ETag: "88ed2808e06e055c6f5acd795e65d8ff"
> Accept-Ranges: bytes
> Content-Type: binary/octet-stream
> Content-Length: 4068
> Server: AmazonS3
Reporter | ||
Comment 4•9 years ago
|
||
(In reply to Jeremy Orem [:oremj] from comment #3) > Hmm, it's working for me > > curl -I http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-i686/te/firefox-40.0b9.checksums > > HTTP/1.1 200 OK > > x-amz-id-2: kZB+uU70UAoKYlf4O27Em5Ll4dye7hwgkPAh6hIKr6TZbUmtZa5y74e4yV4pHkuZ4WG8dFxd+f4= > > x-amz-request-id: F57EA593EEDF6237 > > Date: Fri, 07 Aug 2015 20:20:09 GMT > > x-amz-version-id: jorWBVA4iwZq4LVJnAl_uMPXNCp5T7gG > > Last-Modified: Fri, 07 Aug 2015 16:07:22 GMT > > ETag: "88ed2808e06e055c6f5acd795e65d8ff" > > Accept-Ranges: bytes > > Content-Type: binary/octet-stream > > Content-Length: 4068 > > Server: AmazonS3 Looks like the difference is that Boto appends versionId as a query arg: DEBUG:boto:Path: /pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY If I add that to a curl request, I get a 403: curl "http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY" <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B73686F8406D0C90</RequestId><HostId>thMxCXxo7vcA99TPNBvzR10HSNTGy4B3K6U6NltmZF+ppFz6lOIC083zq4tj35syBCTBZZ5CloY=</HostId></Error>%
Comment 5•9 years ago
|
||
I added an allow on GetObjectVersion curl -I "http://net-mozaws-stage-delivery-firefox.s3.amazonaws.com/pub/firefox/nightly/40.0b9-candidates/build1/linux-x86_64/eo/firefox-40.0b9.checksums?versionId=dH1FAUCOdaUW1.CwfbQUpB12W52idKbY" HTTP/1.1 200 OK x-amz-id-2: kiswmXN8FRncReqvfjVW9PY0/uYOSMEOEbX3ST/dYSKAYtXb/wUXb5PXWixNxDG1 x-amz-request-id: 0448593F0C55C9F8 Date: Mon, 10 Aug 2015 15:54:38 GMT x-amz-version-id: dH1FAUCOdaUW1.CwfbQUpB12W52idKbY Last-Modified: Fri, 07 Aug 2015 16:14:23 GMT ETag: "f124f465aefc0a8cc68ec827ca7475e5" Accept-Ranges: bytes Content-Type: binary/octet-stream Content-Length: 4122 Server: AmazonS3
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•