Investigate using s3 inventory for the data platform

RESOLVED INCOMPLETE

Status

P3
normal
RESOLVED INCOMPLETE
a year ago
a year ago

People

(Reporter: whd, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

a year ago
Last year, Amazon announced some new S3 features that we might find useful. In particular, it might be faster and cheaper to use s3 inventory and flat files than to use s3 list operations in the Dataset APIs. See https://aws.amazon.com/about-aws/whats-new/2016/11/revolutionizing-s3-storage-management-with-4-new-features/ and http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html for starting points. The listings would always be out of date, but we could have a hybrid approach where we list today's data and use the inventory for older data.
Closing abandoned bugs in this product per https://bugzilla.mozilla.org/show_bug.cgi?id=1337972
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.