Closed Bug 1105554 Opened 11 years ago Closed 11 years ago

Daily Aggregate Compression in Redshift

Categories

(Content Services Graveyard :: Tiles: Data Processing, defect)

Product:
Content Services Graveyard
Component:
Tiles: Data Processing
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal
Points:
3

Tracking

(Not tracked)

Status:
RESOLVED INVALID
Iteration:
36.3

People

(Reporter: Mardak, Assigned: tspurway)

References

Details

(Whiteboard: .008)

Copied from https://github.com/tspurway/infernyx/issues/10 Right now we write data to Redshift quite often. This gives us great latency (freshness), but also increases storage requirements due to low aggregation. This increases the price of the Redshift instance as well as reduce query performance. We can introduce a daily job in Infernyx that 'compresses' the data by writing a single row for each unique key per day, and cleaning out the existing day's data.
Depends on: 1105555
Iteration: 37.1 → 37.2
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INVALID
Iteration: 37.2 → 36.3
Points: --- → 3
Whiteboard: .? → .008
Assignee: nobody → tspurway
You need to log in before you can comment on or make changes to this bug.