1285953 - improve rule caching

Status: RESOLVED WONTFIX

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[[e-mail] (:bhearsum)]

We did an initial implementation of this in bug 1111032, but it was a bit of a rush job. Ac ouple ideas for improving it (from https://bugzilla.mozilla.org/show_bug.cgi?id=1111032#c5):
* Fully cache rules table instead of querying it for unique combination of certain parts of the updateQuery.
* Use a local memcache on each node. This would reduce memory usage by having a shared cache across all procs as well as reduce the frequency of blob cache rebuilding (we currently kill procs after 5000 requests, which is every few min, and the blob cache will normally last for up to an hour).

There's probably other options as well.

Comment 1

[Allan [:asilva]]

Are this cache improvement still needed?
By memcache, do you mean a memory database like Memcached or Redis? If is it, why a memcache per node, instead a memcache for all nodes?

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Allan [:ex-dev] from comment #1)
> Are this cache improvement still needed?
> By memcache, do you mean a memory database like Memcached or Redis? If is
> it, why a memcache per node, instead a memcache for all nodes?

It would still be good to improve this, yeah. A shared memcache or redis might be good. It's probably something we should discuss with our operations team before moving forward...

Comment 3

[Allan [:asilva]]

Great, when I finish the "Public API" bug, I can get this, if all is OK to the operations team.
Here at my work we use the Redis shared with six instances, works fine. The negative point is the cost.

Have the rules table more than 2GB in production?

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Allan [:ex-dev] from comment #3)
> Great, when I finish the "Public API" bug, I can get this, if all is OK to
> the operations team.
> Here at my work we use the Redis shared with six instances, works fine. The
> negative point is the cost.
> 
> Have the rules table more than 2GB in production?

It looks like the rules tables is less than 1MB. We mostly cache it because of the overhead of making so many queries, not the size.

Let's talk about this more, and include the Ops team in the discussion, once you're done the public API. I want to make sure we take the right approach here.

Comment 5

[[e-mail] (:bhearsum)]

This hasn't come up again since this bug was filed. We'll file something new as a Github Issue if we want to make changes here.