Currently, if the ES cluster needs to be rebuilt, there is a script that will feed the ES cluster from S3. It is currently manually triggered based on how far back we want to go. It is blunt tool; often blocking current ingestion for hours at a time. Alter the script to be able to run via cron; have it only add more work when the work queue is (mostly) empty.
This script is not just good for rebuilding the ES index, but also good for loading the whole year's test results. If we want to store long time series, then we should have the staging system filled with the 10billion records so we can experience the infrastructure problems at that volume before we go to production.