Daemon to backfill test results into ES

NEW
Unassigned

Status

Testing
ActiveData
2 years ago
2 years ago

People

(Reporter: ekyle, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
Currently, if the ES cluster needs to be rebuilt, there is a script that will feed the ES cluster from S3.  It is currently manually triggered based on how far back we want to go.  It is blunt tool; often blocking current ingestion for hours at a time.

Alter the script to be able to run via cron; have it only add more work when the work queue is (mostly) empty.
(Reporter)

Updated

2 years ago
Assignee: klahnakoski → nobody
(Reporter)

Updated

2 years ago
Blocks: 1193249
(Reporter)

Comment 1

2 years ago
This script is not just good for rebuilding the ES index, but also good for loading the whole year's test results.  If we want to store long time series, then we should have the staging system filled with the 10billion records so we can experience the infrastructure problems at that volume before we go to production.
You need to log in before you can comment on or make changes to this bug.