1300082 - Followups to the Memcachier stunnel solution

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

I'm not overly happy with the outcome of bug 1291307 - that was just the best of all of the suboptimal alternatives.

Once we're on Heroku, some options to consider:

* Fixing some of the deficiencies in memcachier-tls-buildpack, such as:
  - https://github.com/memcachier/memcachier-tls-buildpack/issues/11
  - https://github.com/memcachier/memcachier-tls-buildpack/issues/12
* Switching from memcached to Redis, since:
  - it's generally considered less of a legacy technology
  - Heroku has a native Redis addon
  - Heroku provides (and maintains) an stunnel buildpack for Redis of their own (https://github.com/heroku/heroku-buildpack-redis) that's actually written properly
  - some Redis clients even have native TLS support
* Revisiting the viability of adding TLS support to eg python-binary-memcached (bmemcached) given that the custom Memcachier server implementation negates the consistent hashing concern in bug 1291307 comment 1. Though this still means switching from a C-backed client to a pure Python one (with perf impact; though opens us to using gevent).

Comment 1

[Ed Morley [:emorley]]

> * Switching from memcached to Redis, since:

Comparison between Redis providers/thoughts to consider at:
https://redislabs.com/redis-comparison
https://redislabs.com/blog/six-things-to-consider-when-using-redis-on-heroku

The two best options appear to be (both support SSL):
https://elements.heroku.com/addons/heroku-redis
https://elements.heroku.com/addons/rediscloud

Comment 2

[Ed Morley [:emorley]]

Whilst it's not clear whether we will/won't want to switch to Redis yet, I've done some initial research to at least get the ball rolling on some issues/tickets.

Redis Cloud TLS is a paid-for addon, that requires contacting support, so I've not been able to test it.

I've been able to test Heroku Redis' TLS (since it comes for free with any prod plans). It currently uses self-signed certs, REDIS_URL could be improved, and the docs were missing some details, so I've filed:
https://help.heroku.com/tickets/396267
https://help.heroku.com/tickets/396276
https://help.heroku.com/tickets/396297
https://github.com/heroku/heroku-buildpack-redis/issues/15
https://github.com/heroku/heroku-buildpack-redis/issues/16

redis-py (the recommended Python redis-client) has no TLS docs, so I've filed:
https://github.com/andymccurdy/redis-py/issues/780

It's also not clear which is better out of the two Django backends that make use of redis-py, so I've asked them to add comparisons:
https://github.com/niwinz/django-redis/issues/217
https://github.com/sebleier/django-redis-cache/issues/121

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #0)
> * Fixing some of the deficiencies in memcachier-tls-buildpack, such as:

I've dug into this option a bit, since one of the ways to fix bug 1381151 was to just turn memcachier-tls-buildpack into a fork of heroku-buildpack-redis (which already does have Heroku-16 support). See:
https://github.com/memcachier/memcachier-tls-buildpack/issues/11#issuecomment-315471002

This looks promising, though there are a few upstream issues I'd like to see fixed before I start forking heroku-buildpack-redis into memcachier-tls-buildpack, for which I've opened PRs / investigated:
https://github.com/heroku/heroku-buildpack-redis/pull/21
https://github.com/heroku/heroku-buildpack-redis/pull/22
https://github.com/heroku/heroku-buildpack-redis/pull/23
https://github.com/heroku/heroku-buildpack-redis/issues/13

Comment 4

[Ed Morley [:emorley]]

Wontfix since we're switching to Redis in bug 1384518.