Closed Bug 792931 Opened 12 years ago Closed 11 years ago

Plan best way to cache elastic search

Categories

(Marketplace Graveyard :: Code Quality, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: andy+bugzilla, Assigned: alexis+bugs)

References

Details

(Whiteboard: p=0)

If the load test in bug 792929 shows an improvement, come up with a plan on how to cache elastic search. Some of the things discussed are:

- some of the libraries are trying to do caching, let's see if that is something we want to use.
- make a proxy to pool and cache connections.
- something marketplace specific.
- something else.

Once we've got a plan, we'll file more bugs.
I don't think we should do caching in the form of a network proxy. Having something handling this in memory (via some sort of python code) sounds a better option than having a separate process handling this, but I would be happy to be proved wrong.

The first thing I'll be doing on this is to have a look at what these requests are, if they can be factorized or if they should be cached server-side rather than reaching ES at each request.
Assignee: nobody → alexis
Would that be in memory for each process/webhead? We'd have the same data in lots of different caches? We have got this super memcached available to us now...
Oh, I'm of course thinking about shared memory, why not in the form of memcache, especially if we have it available and waiting for things to store!
Is the bottleneck the actual *number* of queries, or the fact that the queries tend to take time to execute?

Are we trying to reduce the number of queries to ES or are we trying to make search faster on the ES-side?

I'm asking this because I don't know how ES is configured at the moment, and it seems that the configuration of it can influence on the performances.

I assume that we're trying to reduce the number of queries we're doing on the ES servers, but I want to make sure I'm not digging in the wrong direction, and also that the "improve the settings in ES" direction is taken care of.
Elastic Search is our bottleneck for a browsing usage as each ES node eats the whole CPU on high load

I assume what we want is reduce the queries made on ES by using Membase to cache anonymous results.

We need a smart invalidation system though since adding/removing web app etc will change the results. 

I suspect we should cache the most common queries that are used for browsing at first. and see how it goes
We are wrapping our ES calls with statsd if you want to get a sense how long they are taking...
https://github.com/mozilla/zamboni/blob/master/apps/amo/search.py#L193

I suspect for the homepage a default timeout (cache-machine uses 5 minutes?) on these would be fine. Enough to keep some load off ES but not long enough for changes to not come through.
Thanks for the pointer.

However, the code :robhudson pointed out doesn't seem to be used for every call we're doing to ES. For instance, the calls to ES here https://github.com/mozilla/zamboni/blob/master/mkt/webapps/models.py#L555 aren't made with the client that's in apps.amo.search but directly with ElasticUtils, unless I'm missing something.

Is it something intended? If not, I probably can make all ES calls go through statsd, maybe by adding statsd support to EU.

Adding cache on the EU side seems reasonable to me, but means that we will need to opt-out the requests we don't want to cache explicitly. I'm thinking about naming all the requests we are doing, so we can invalidate their cache easily. I'm not sure this request-naming idea is compatible with having cache at the EU level.

The other approach is to cache each of the calls at a higher level, and to deal with cache invalidation at this same level, like what's proposed by :willkg on bug 792927. This seems to be the most straightforward approach, but I don't know if that's the best way to cache everything. The advantage of caching everything in EU is that it will be cached automatically without further work on the views side.

Because some of you already dealt with caching, and especially ES caching, I would be interested to have your opinions on this. Do you have any concerns / ideas?
(In reply to Alexis Metaireau (:alexis) from comment #7)
> However, the code :robhudson pointed out doesn't seem to be used for every
> call we're doing to ES. For instance, the calls to ES here
> https://github.com/mozilla/zamboni/blob/master/mkt/webapps/models.py#L555
> aren't made with the client that's in apps.amo.search but directly with
> ElasticUtils, unless I'm missing something.
> 
> Is it something intended? If not, I probably can make all ES calls go
> through statsd, maybe by adding statsd support to EU.

Oh right. Preferably we'd convert our calls to using EU and get rid of our custom elasticsearch wrapper. If I've got it right, that custom wrapper was the initial code that got EU started and since EU has gone further with better/more support.
Okay, that's a separate issue then, created bug 795944 about it.
Whiteboard: p=0
I don't think this is relevant for the moment. Please reopen if you disagree.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.