851147 - Load test ES

Status: RESOLVED FIXED

(Marketplace Graveyard :: Search, defect, P1)

Description

[Wil Clouser [:clouserw]]

We need to get a sense of how ES can scale with the plan of having 32+ binary flags along with our other use filters.  This doesn't need to be super detailed but think from a high level perspective:

- How many users can we support with how many apps?  What happens with 2x and 4x the apps/users?

I'm just looking for a sense of how many servers we need now and how many servers we need in a year.

Comment 1

[Hanno Schlichting]

Some numbers for reference from current prod:

- about 360k add-ons/apps
- each add-on entry already has ~100 keys 
- the JSON blob for each add-on is ~3kb each

So in terms of adding 32 binary fields, we don't actually get a huge increase in data size.

As for longer term capacity planning:

- Move all stats into monolith (two separate deployments for amo/mkt with at least 3 prod ES servers each).
- After enabling compression, the entire addons index (including collections, users, ...) is likely ~3.5gb in total, with 4 replicas: 17.5gb.
- Currently we have 6 servers with ~36gb RAM each

So after compression and stats are out, each server will hold the entire index in memory and use just half of the RAM.

If I'd have to take an educated guess, I'd say six servers will be plenty to hold the next couple of hundred of thousands of apps. As for handling more users, that's a lot more complicated to answer, as this involves the entire stack. But I'd bet ES isn't going to be the bottleneck for some time.

So for long term planning, decisions on multi-dc deployments (3 or 5 servers per DC) will likely have a bigger impact. And if we should split amo/mkt ES into separate clusters, we'd reduce the mkt data size to almost nothing. At least splitting them into separate indexes in the same cluster would help with cache efficiencies.

And on the upside: We are currently using ES 0.19 based on Lucene 3.6. In a couple of months we can likely upgrade to later ES versions based on Lucence 4.x. That should give us a lot of performance improvements and memory savings. Depending on the exact metric, there's improvements in the range of 20% to a factor of 100 and more. So time is working for us here.

Comment 2

[Wil Clouser [:clouserw]]

Awesome, thanks.  I didn't realize we already had so many keys on the apps, it sounds like we can close this bug.  Cheers.