Closed Bug 1357270 Opened 7 years ago Closed 7 years ago

Investigate using s3 inventory for the data platform

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, enhancement, P3)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: whd, Unassigned)

References

Details

Last year, Amazon announced some new S3 features that we might find useful. In particular, it might be faster and cheaper to use s3 inventory and flat files than to use s3 list operations in the Dataset APIs. See https://aws.amazon.com/about-aws/whats-new/2016/11/revolutionizing-s3-storage-management-with-4-new-features/ and http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html for starting points. The listings would always be out of date, but we could have a hybrid approach where we list today's data and use the inventory for older data.
Closing abandoned bugs in this product per https://bugzilla.mozilla.org/show_bug.cgi?id=1337972
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.