947278 - [traceback] MemcachedKeyLengthError: Key length is > 250

Status: RESOLVED FIXED

(Input Graveyard :: Submission, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Traceback (most recent call last):

  File "/data/www/input.mozilla.org/input/vendor/lib/python/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/usr/lib64/python2.6/site-packages/newrelic-1.11.0.55/newrelic/api/object_wrapper.py", line 216, in __call__
    self._nr_instance, args, kwargs)

  File "/usr/lib64/python2.6/site-packages/newrelic-1.11.0.55/newrelic/hooks/framework_django.py", line 475, in wrapper
    return wrapped(*args, **kwargs)

  File "/data/www/input.mozilla.org/input/vendor/lib/python/django/views/decorators/csrf.py", line 77, in wrapped_view
    return view_func(*args, **kwargs)

  File "/data/www/input.mozilla.org/input/vendor/lib/python/django/views/decorators/cache.py", line 89, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)

  File "/data/www/input.mozilla.org/input/fjord/feedback/views.py", line 298, in feedback_router
    return view(request, *args, **kwargs)

  File "/data/www/input.mozilla.org/input/fjord/feedback/views.py", line 60, in _requires_firefox
    return func(request, *args, **kwargs)

  File "/data/www/input.mozilla.org/input/vendor/lib/python/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)

  File "/data/www/input.mozilla.org/input/fjord/feedback/views.py", line 169, in desktop_stable_feedback
    response, form = _handle_feedback_post(request)

  File "/data/www/input.mozilla.org/input/fjord/feedback/utils.py", line 70, in _wrapped
    field=None, rate=rate, keys=keyfun)

  File "/data/www/input.mozilla.org/input/vendor-local/src/django-ratelimit/ratelimit/helpers.py", line 82, in is_ratelimited
    counts = _incr(cache, _keys, period)

  File "/data/www/input.mozilla.org/input/vendor-local/src/django-ratelimit/ratelimit/helpers.py", line 62, in _incr
    counts = cache.get_many(keys)

  File "/data/www/input.mozilla.org/input/vendor/lib/python/django/core/cache/backends/memcached.py", line 79, in get_many
    ret = self._cache.get_multi(new_keys)

  File "/usr/lib64/python2.6/site-packages/newrelic-1.11.0.55/newrelic/api/memcache_trace.py", line 73, in __call__
    return self._nr_next_object(*args, **kwargs)

  File "/data/www/input.mozilla.org/input/vendor-local/packages/python-memcached/memcache.py", line 911, in get_multi
    server_keys, prefixed_to_orig_key = self._map_and_prefix_keys(keys, key_prefix)

  File "/data/www/input.mozilla.org/input/vendor-local/packages/python-memcached/memcache.py", line 625, in _map_and_prefix_keys
    self.check_key(str_orig_key, key_extra_len=key_extra_len)

  File "/data/www/input.mozilla.org/input/vendor-local/packages/python-memcached/memcache.py", line 1023, in check_key
    % self.server_max_key_length)

MemcachedKeyLengthError: Key length is > 250


POST:<QueryDict: {u'description': [u'\u5347\u7ea7\u4e86\u65b0\u7248\u672c\u4e4b\u540e\u7684\u6807\u7b7e\u9875\u8bbe\u7f6e\u9879\u76ee\u53d8\u5c11\u4e86\uff0c\u4e66\u7b7e\u6253\u5f00\u4f1a\u66ff\u4ee3\u76ee\u524d\u7684\u6807\u7b7e\u9875\uff1b\u6807\u7b7e\u9875\u4e5f\u4e0d\u80fd\u518d\u8bbe\u7f6e\u5173\u95ed\u6807\u7b7e\u9875\u540e\u8df3\u8f6c\u5230\u54ea\u4e2a\u6807\u7b7e\u9875\u3002\u4e3a\u4ec0\u4e48\u5347\u7ea7\u4e86\u529f\u80fd\u53cd\u800c\u53d8\u5c11\u4e86\uff1f'], u'url': [u''], u'email_ok': [u'on'], u'device': [u''], u'csrfmiddlewaretoken': [u'wBc5PnhlMt1dgSZnxRpn7rFWHCa7aNiO'], u'manufacturer': [u''], u'happy': [u'0', u'0']}>,


Pretty sure I need to tweak how the key is generated and that'll fix this.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Saw another one go by today. Grabbing this to work on it now.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I spent some time trying to reproduce this with the data from the traceback, but wasn't able to which puzzles me.

Anyhow, I added a test and tweaked the algorithm so it might handle the case that causes this issue better.

PR in https://github.com/mozilla/fjord/pull/188

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master in https://github.com/mozilla/fjord/commit/37792ed

Mike pointed out that the problem was probably a mediocre algorithm coupled with an ipv6 address. So this fix fixes the mediocre algorithm and accounts for ipv6 addresses.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pushed to prod just now.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Reopening this because I just saw another one go by after I had pushed this to production. I'm really puzzled, though.

Regardless, I discovered Django has a cache key generator. I'll switch to use that.

https://docs.djangoproject.com/en/1.3/topics/cache/#cache-key-transformation

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pretty sure I know what the problem is. The previous algorithms got us a key that was pretty close, but always under the 250 character max. However, I forgot that stage and prod environments tack on an additional key prefix which is probably bumping us over when the user has an ipv6 address.

New fix in a PR: https://github.com/mozilla/fjord/pull/192

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master in https://github.com/mozilla/fjord/commit/45e6c5c

Pushed to prod just now. Looks fixed, but I'll keep an eye on it.