1218970 - Redshift cluster for Push metrics

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Rob Miller [:rmiller]]

Some of the requested Push metrics can be extracted from the Push and Endpoint server logs, but require too much memory for the state to be maintained in a Heka or Hindsight Lua sandbox. Redshift seems likely to be a reasonable and cost-efficient solution for storing this data, so I'd like to get access to a Redshift cluster to experiment. The most minimal setup possible is fine for initial testing.

Comment 1

[Daniel Thorn [:relud]]

for development go ahead and use your privileges in the dev account to spin up a new single-node cluster. Note that redshift's underlying architecture is designed for dumping large numbers of rows into a table, and not updating those rows. If there are going to be a large number of small transactions (a few rows updated in small tables), or if the data is going to be updated instead of added to, then postgres is likely a better solution.

Comment 2

[Rob Miller [:rmiller]]

I assume this means I'd need to use an EC2 instance in the dev IAM to talk to it, and not one of the "ad-hoc analysis worker" VMs available through the http://analysis.telemetry.mozilla.org/ site, right? I'll give that a shot. I've assigned this bug to myself. If I can get things bootstrapped, I'll come back and close it. If and when we get to a point where we need an ops-managed cluster, I'll open a new one.

Thanks for your help!