1431497 - Switch Django, KumaScript caching to Redis

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, enhancement, P2)

Description

[John Whitlock [:jwhitlock]]

What feature should be changed? Please provide the URL of the feature if possible.
==================================================================================
Kuma uses three caches:

- Local memory is used as the default cache. This is private to each host, and is cleared on restarts
- Memcached is used as a general Django cache. The big users are Django Cacheback, Constance, and Ratelimiting
- Redis is used (starting in AWS) as the Celery task queues and result store.

KumaScript also has the ability to use memcache.

We should move the general cache to Redis, using database 3 for namespacing.

What problems would this solve?
===============================
In general, Redis is a better caching engine. See https://stackoverflow.com/a/11257333/10612

Switching to a single caching engine would save on AWS rental and lower maintenance and monitoring efforts. We need Redis's features for Celery, so it is the only choice for a single cache engine.

We've hit memcache's limitations in the page. memcache is limited to 1 MB values, simple string storage, and does not allow introspection of keys. Redis provides all the features of memcached, plus additional data storage types for specific workloads.

Who would use this?
===================
Staff developers and SREs.

What would users see?
=====================
A single cache engine for Kuma development and deployment

What would users do? What would happen as a result?
===================================================
Everyday usage would be the same, but advanced usage and troubleshooting will be possible.

Is there anything else we should know?
======================================
"memcache" is used many times in the Kuma codebase. This will need to be generalized so that switching the engine doesn't break code.

KumaScript is hardcoded to use memcache as well. A Redis client will need to be added. There may be an abstract library similar to Django's caching interface.

The production Redis server should work for the additional caching load. The deployments will need to be configured to switch engines when the code is ready for the switch.

Comment 1

[[github robot]]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/7ed89daccfa3365af6336b43139fc36683cb4df7
bug 1431497: Migrate from memcache to redis (#4870)

Combine the two cache configurations (local memory and memcache) into a single cache configuration based on Redis. This also removes the memcache_hashring library, which doesn't support Python 3.

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/1f4ed0efed6276fac722b77282d25dc41fd46b81
bug 1431497: Swap spam for health in noext tests

The spam tests use waffle flags and switches, which are now stored in
the Redis cache, and fail in the noext tests. Instead, check
kuma/health.

https://github.com/mozilla/kuma/commit/f7001b537e1e103cde563c5dde4c9440f332d70f
Merge pull request #4914 from jwhitlock/noext-1431497

bug 1431497: Swap spam for health in noext tests

Comment 3

[John Whitlock [:jwhitlock]]

KumaScript switched to Redia as part of https://github.com/mdn/kumascript/pull/1035.