1308532 - Proposal: Presto DB Running on a single X1 Instance

Status: RESOLVED WONTFIX

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P3)

Description

[Frank Bertsch [:frank]]

CURRENT:
33 m4.xlarge
- .299$ an hour per instance * 33 = 9.867$ 
- 528 GiB
- 132 vCPU

PROPOSE:
1 X1
- $13.338 On Demand
- $7.671 Hourly 1-Year Term
- 1952 GiB
- 128 vCPU

With 4x the memory and no network overhead, not only would queries by lightning fast, but we would reduce the need for resources. More people would be able to use that machine than use the similarly priced cluster. Whole datasets could be cached in memory.

For example, a single version of longitudinal takes up ~275 GiB. We could cache the most recent version, and probably a few other datasets (client_count, xSec, etc.), and have lightning fast queries for the majority of our users.

The downside is this does not run on EMR, so it would probably be more work on the Ops side.

Comment 1

[Frank Bertsch [:frank]]

Blake mentioned that he's looked into this, and it may not be possible. There's very little information online about running Presto on an X1 instance. I'm pinging robotblake to see what he's found.

Comment 2

[Blake Imsland [:robotblake]]

The JVM doesn't scale particularly well past 32GB so we're going to skip this, also with Athena being a viable option for the heavy queries.