881405 - Stop using cache-machine

Status: RESOLVED FIXED

(Marketplace Graveyard :: Code Quality, defect, P3)

Description

[Rob Hudson [:robhudson]]

I believe we can stop using cache-machine...

1. It's the last thing getting us off of redis (bug 776156)

2. We've killed the use of all consumer pages in place of an API which can be easily cached manually either by using Django's low level caching directly or using front-end caching or both.

3. Devhub and reviewer tools and admin pages want to operate on the latest data and caching often introduces bugs or confusion. We can likely survive without caching in these areas. Those areas where we do need caching we can cache manually also.

Let's discuss.

Comment 1

[Andy McKay]

+1 for the marketplace

But I would rather make sure we are going nice and fast before turning this off, just in case.

I think we'd need to leave AMO with Cache Machine on, so how would we remove it?

Comment 2

[Rob Hudson [:robhudson]]

(In reply to Andy McKay [:andym] from comment #1)
> I think we'd need to leave AMO with Cache Machine on, so how would we remove
> it?

Oh right... both use the same models which inherit cache-machine. Hmm. We'd have to get creative to accomplish that.

Comment 3

[Christopher Van Wiemeersch [:cvan]]

I think bug 869741 would make this easy :)

Comment 4

[Rob Hudson [:robhudson]]

One idea:

Write a middleware that wraps the `process_response` with skip_cache:
https://github.com/mozilla/zamboni/blob/master/apps/amo/models.py#L32-L40

This affects the model's parent class to add a `no_cache` which bypasses cache-machines machinery. It might work and easily separates mkt from amo.

Comment 5

[Wil Clouser [:clouserw]]

According to https://groups.google.com/forum/#!topic/mozilla.dev.marketplace/W3YkWY6LOCg jezdez may have some good advice. He got MDN off of cachemachine earlier this year:

https://github.com/mozilla/kuma/pull/1853
https://github.com/jezdez/django-memcached-hashring

Comment 6

[Mathieu Pillard [:mat]]

A new setting to enable/disable was introduced in https://github.com/mozilla/zamboni/commit/87b17584046b6f1e462e35b015708c3375cc4c8c

It's set to False by default, but True on prod ; this will enable us to test what it's like to not have cache-machine on dev/stage/locally for a while, while keeping prod as before. Then we'll figure out what the next steps are.

Comment 7

[Mathieu Pillard [:mat]]

Two views that are hit relatively often and will probably need simple caching are mkt.detail.views.manifest and mkt.downloads.views.download_file. Both are fairly simple (and the manifest already uses caching) but loads Webapp and File models, which can be fairly costly.

Comment 8

[Mathieu Pillard [:mat]]

cache-machine is disabled on dev/stage/locally but we never figured out a way / found the time to load test everything. Unassigning myself for now.

Comment 9

[Mathieu Pillard [:mat]]

Performance problems without cache-machine enabled are now solved, we are turning it off in production next push. 

Relevant IRC discussion:

<mat> jason: from what I can see in the latest newrelic logs (I find it hard to compare time ranges) performance for SearchView on stage during the log replay is now pretty good without cache-machine - I removed/better cached a few queries and it shows
<jason> mat: sweet
<jason> do you think we are ready to deploy to prod?
* jason also looks at newrelic
<jason> based on the 1 month view it looks like it is performing better than before with cache machine
<mat> jason: I was thinking of disabling cache-machine for next push in prod, see how it goes
 and if there are no issues for a week or two, completely remove it (which will be a big commit, lots of small things to remove, like all our calls to .no_cache())
<jason> mat: that works for me, let's do it

Comment 10

[Mathieu Pillard [:mat]]

cache-machine deactivated for prod in https://github.com/mozilla/zamboni/commit/7fff1884aefedc6bc0713ecabc6506fc4f01164a

Comment 11

[Mathieu Pillard [:mat]]

We've been running without cache-machine for a week now. Things look mostly good in new relic. Some highlights from the most time consuming views:
- MultiSearchView is virtually the same as before.
- manifest view is also virtually the same. More time is spent talking to the database but it's compensated elsewhere.
- AppViewSet is slightly worse, mostly due to price currency queries.
- RatingViewSet is slightly worse, mostly due to price currency queries (again).
- commonplace view is virtually the same as before.
- download_file view took a big perf hit.

I think we can go ahead with the complete removal (providing we fix bug 952550) but we need to fix download_file view perf regression and fix the price currency queries once and for all (bug 1089848)

Comment 12

[Mathieu Pillard [:mat]]

Fixed in https://github.com/mozilla/zamboni/commit/f55c1ff3dc3d8ae10027b2aa8dbab98ae10d1b98


STR:
- None, this can affect everything in the API / Developer Pages / Reviewer Pages. However, this is just a cleanup, cache-machine had already been deactivated, so the risk is somewhat low.