[traceback] ServerError: error 10 from memcached_add: SERVER ERROR

VERIFIED FIXED in 6.2.7

Status

P3
major
VERIFIED FIXED
7 years ago
3 years ago

People

(Reporter: stephend, Assigned: robhudson)

Tracking

unspecified
6.2.7
Dependency tree / graph

Details

(Whiteboard: [pylibmc], URL)

http://addons-dev.allizom.org/en-US/firefox/language-tools/:

Traceback (most recent call last):

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/addons-dev.allizom.org/zamboni/apps/browse/views.py", line 139, in language_tools
    'search_cat': '%s,0' % amo.ADDON_DICT})

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/jingo/jingo/__init__.py", line 79, in render
    rendered = render_to_string(request, template, context)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/jingo/jingo/__init__.py", line 97, in render_to_string
    return template.render(**get_context())

  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)

  File "/data/www/addons-dev.allizom.org/zamboni/apps/browse/templates/browse/language_tools.html", line 19, in top-level template code
    <br>({{ addon.locale_disambiguation }})

  File "/data/www/addons-dev.allizom.org/zamboni/templates/base.html", line 111, in top-level template code
    {% block main_content %}

  File "/data/www/addons-dev.allizom.org/zamboni/templates/base.html", line 116, in block "main_content"
    {% block content %}{% endblock %}

  File "/data/www/addons-dev.allizom.org/zamboni/apps/browse/templates/browse/language_tools.html", line 26, in block "content"
    {% cache addons %}

  File "/data/www/addons-dev.allizom.org/zamboni/apps/amo/ext.py", line 22, in _cache_support
    return sup(key, obj, timeout, extra, caller)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django-cache-machine/caching/ext.py", line 74, in _cache_support
    return caching.base.cached_with(obj, caller, key, timeout)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django-cache-machine/caching/base.py", line 284, in cached_with
    return cached(f, key, timeout)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django-cache-machine/caching/base.py", line 264, in cached
    val = function()

  File "/data/www/addons-dev.allizom.org/zamboni/apps/browse/templates/browse/language_tools.html", line 44, in template
    {% for lang, locale in locales %}

  File "/data/www/addons-dev.allizom.org/zamboni/apps/browse/views.py", line 97, in _get_locales
    for addon in addons:

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django/django/db/models/query.py", line 107, in _result_iter
    self._fill_cache()

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django/django/db/models/query.py", line 784, in _fill_cache
    self._result_cache.append(self._iter.next())

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django-cache-machine/caching/base.py", line 108, in __iter__
    self.cache_objects(to_cache)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django-cache-machine/caching/base.py", line 115, in cache_objects
    cache.add(query_key, objects, timeout=self.timeout)

  File "/data/www/addons-dev.allizom.org/zamboni/vendor/src/django/django/core/cache/backends/memcached.py", line 54, in add
    return self._cache.add(key, value, self._get_memcache_timeout(timeout))

ServerError: error 10 from memcached_add: SERVER ERROR

Updated

7 years ago
Assignee: nobody → jbalogh
Whiteboard: [pylibmc]
Target Milestone: --- → 6.2.3

Updated

7 years ago
Blocks: 683092

Updated

7 years ago
Duplicate of this bug: 683092
Target Milestone: 6.2.3 → Q3 2011

Updated

7 years ago
Blocks: 684300
Priority: -- → P1
Target Milestone: Q3 2011 → 6.2.5

Updated

7 years ago
Duplicate of this bug: 689076
This happens when we try to shove more than 1MB into memcached (that's the limit). It only happens in -dev because we're testing out pylibmc (a new memcached connector).

We should probably enable gzip to avoid this for large payloads (the cpus in phx are bored), along with subclassing django's class to catch and log these errors (more parentheticals).
Assignee: jbalogh → chudson
thanks jeff.  QA: This is from the new library we're experimenting with on -dev.  It won't manifest in production.
Priority: P1 → P3

Comment 5

7 years ago
Adding in logging of attempted large writes to memcached would be a nice addition as jeff said.
(Assignee)

Comment 6

7 years ago
I found pylibmc has zlib support already if you pass it the argument on `set` and `add`.  I sent a pull request last night but willing to do more (like not hard code the min size?), or change the approach.  Feedback wanted.  :)
https://github.com/jbalogh/django-pylibmc/pull/1

I thought Jeff already added logging for this?
https://github.com/jbalogh/django-pylibmc/commit/9f59ddf7d686c8dcc2fec3fa5f9d55c86eb56e74

Or is there other logging that's wanted?
We're not using django-pylibmc right now. That was an experiment from before django had pylibmc support in trunk, and it never went far because we had trouble with pylibmc.

That lib probably needs to be updated to match the current cache internals.
Target Milestone: 6.2.5 → 6.2.6

Updated

7 years ago
Target Milestone: 6.2.6 → 6.2.7
This is causing problems with the automated tests as well.  Glad to see that it's on the list for a push next week.
(Assignee)

Comment 10

7 years ago
I pushed the updated django-pylibmc library: https://github.com/jbalogh/django-pylibmc/commit/5cf5130

I think what's left is to ask Jeff to push that to PyPI and get it installed on -dev for testing?
(In reply to Rob Hudson [:robhudson] from comment #10)
> I pushed the updated django-pylibmc library:
> https://github.com/jbalogh/django-pylibmc/commit/5cf5130
> 
> I think what's left is to ask Jeff to push that to PyPI and get it installed
> on -dev for testing?

Let's just get it in vendor/src for now.
Depends on: 694604
(Assignee)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Verified FIXED; thanks, Rob:

[11:53:46.865] GET https://addons-dev.allizom.org/en-US/firefox/language-tools/ [HTTP/1.1 200 OK 138ms]
Status: RESOLVED → VERIFIED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.