Closed Bug 1134217 Opened 9 years ago Closed 9 years ago

Create a fast cache of the files in the S3 Data Store

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mreid, Assigned: rvitillo)

References

Details

Listing the contents of an S3 bucket is quite slow. We should create an index of the contents for filtering / searching the data.

We can use S3 Event notification for keeping the index up to date:
https://aws.amazon.com/blogs/aws/s3-event-notification/
Priority: -- → P2
Assignee: nobody → mreid
Assignee: mreid → nobody
Blocks: 1125451
Assignee: nobody → rvitillo
Blocks: 1152539
Depends on: 1156254
Depends on: 1156325
A new filtering service backed by SimpleDB and updated in real-time with an AWS lambda function has been deployed for v2 submissions. Comparisons with the old index confirms that the new index is working properly. The next step is to add v4 submissions to it.
Depends on: 1158175
Depends on: 1158790
Depends on: 1159312
V4 submissions have been added to the index and are updated in real-time by an AWS lambda function.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.