Putting work items into a queue, and syncing those work items with data in S3 is a minor pain when writing ETL scripts. Specifically, the JSON configuration must be careful to properly name the queue with its related S3 bucket. Make an abstraction, called Permanent Queue, that hides this abstraction and provides all he features of a queue, with durability of S3, plus the ability to reprocess any (previously) queued items.
|
(Reporter) | |
Updated•2 years ago
|
||
Description
•