927086 - Switch to django-memcached-pool on zamboni

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Marketplace, task)

Description

[Jason Thomas [:jason]]

We are starting to become overloaded by memcache connections and need to enable connection pooling. 

Test https://github.com/mozilla/django-memcached-pool/ on -dev to make sure it is working before configuring on production.

Comment 1

[Jeremy Orem [:oremj]]

Were we seeing any tracebacks or just broken responses?

Comment 2

[Jason Thomas [:jason]]

https://addons-dev.allizom.org/services/monitor memcache connection tests were failing stating that there were no configured nodes. I did not see any sentry tracebacks or interesting messages in the logs. Devs reported that the -dev response was slow.

Comment 3

[Jason Thomas [:jason]]

Enabled on -dev again https://github.com/mozilla/zamboni/commit/e39d1ff9aa2a44b1572bc17a3511f5cd45246928. /services/monitor looks like it is working.

Comment 4

[Jason Thomas [:jason]]

Seeing the following traceback http://sentry.dmz.phx1.mozilla.com/addons/addons-dev/group/17312/

Comment 5

[Jeremy Orem [:oremj]]

It looks like this may have been fixed to show an actual memcache error in newer versions of ultramemcached.

Comment 6

[Jeremy Orem [:oremj]]

I've upgraded umemcache on dev from 1.5 to 1.6.3.

Comment 7

[Jason Thomas [:jason]]

Updated django-memcached-pool to 0.5. Afterwards seeing the following tracebacks http://sentry.dmz.phx1.mozilla.com/addons/addons-dev/group/17334/

Comment 8

[Jason Thomas [:jason]]

@tarek or @cvan would you be able to take a look at the traceback in comment 7?

Comment 9

[Christopher Van Wiemeersch [:cvan]]

Allen, I think you were up in this code:

memcachepool.cache in call
TypeError: argument 2 must be string or read-only buffer, not int

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 89, in get_response
    response = middleware_method(request)
  File "newrelic/hooks/framework_django.py", line 214, in wrapper
    return wrapped(*args, **kwargs)
  File "access/middleware.py", line 25, in process_request
    amo_user = RequestUser.objects.get(pk=request.user.pk)
  File "django/db/models/manager.py", line 131, in get
    return self.get_query_set().get(*args, **kwargs)
  File "django/db/models/query.py", line 361, in get
    num = len(clone)
  File "django/db/models/query.py", line 85, in __len__
    self._result_cache = list(self.iterator())
  File "caching/base.py", line 118, in __iter__
    self.cache_objects(to_cache)
  File "caching/base.py", line 125, in cache_objects
    cache.add(query_key, objects, timeout=self.timeout)
  File "memcachepool/cache.py", line 147, in add
    flag)
  File "memcachepool/cache.py", line 51, in call
    raise exc

access/middleware.py ? in process_request (application)
    amo_user = RequestUser.objects.get(pk=request.user.pk)

Comment 10

[Jeremy Orem [:oremj]]

We are going to evaluate twemproxy, which may make this bug unnecessary.

Comment 11

[Tarek Ziadé (:tarek)]

Sorry for the late response. Let me know if you still need support on this

Comment 12

[Jeremy Orem [:oremj]]

Closing this out. It looks like twemproxy will take care of this for us and django will be adding an official library eventually https://code.djangoproject.com/ticket/11331.