782577 - Migrate sync1.1 to python-memcached instead of pylibmc

Status: VERIFIED WONTFIX

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

Attachment: patch migrating to python-memcached from pylibmc

This is a potential alternative to Bug 781727.

In my tests, python-memcached is approximately twice as slow as pylibmc but it does not block the gevent event loop.

When we had the always-very-fast response guarantees of memcached I think pylibmc was clearly the better option.  But with couchbase, we've been seeing that heavy write loads can cause it to start responding very slowly, making pylibmc block for a second or more.  I'd like to evaluate python-memcached in stage to see how it handles this scenario.

Posting for review, but I don't think we should commit it until we've tried a fully-healthy stage setup with the loadtest changes from Bug 782002 and seen what effect that has on couchbase performance.

Comment 1

[Ryan Kelly [:rfkelly]]

Attachment: patch to make memcached tests use memcache

Ugh, forgot to change the import in test_memcachedsql.py to match.

Comment 2

[Ryan Kelly [:rfkelly]]

Committed in:

http://hg.mozilla.org/services/server-storage/rev/f2964668e926
http://hg.mozilla.org/services/server-storage/rev/65e3698f2433

Leaving the bug open while we see whether this helps us in stage.

Comment 3

[James Bonacci [:jbonacci]]

Just FYI - I verified that these changes made it to Stage as part of this bug:
784229

Comment 4

[Ryan Kelly [:rfkelly]]

Initial deployment of this to stage did not go well.  Under modest load we are getting "SERVER ERROR proxy downstream timeout" errors from moxi, which make python-memcache barf in strange ways.

Moxi gives these errors when the downstream couchbase server does not respond within 5s, so it might not be python-memcache's fault.  But I don't think we were seeing these errors with pylibmc.  So more investigation is needed.

We're also still getting some reports that the gevent eventloop is being blocked, although I think they might be spurious.  Bug 786176 adds a better detector which should give us a more definitive answer in that regard.

Comment 5

[James Bonacci [:jbonacci]]

OK. So marking this appropriately, since we, in effect, rolled back.
See bug 786479

Comment 6

[Ryan Kelly [:rfkelly]]

I'm going to re-open this since it is still on default branch but was backed out in the release branch.  We need to remember to back it out on default branch once we're happy that this was the right move.

Comment 7

[Ryan Kelly [:rfkelly]]

We've been having good results from pylibmc with the adjusted funkload suite, and there doesn't seem to be an easy way to work around the lack-of-connection-pooling issues we hit when trying to push this to stage.  So, I'm marking this WONTFIX.

The high-level goal to replace pylibmc will be pursued via umemcache and a custom pooling layer.

Backed out on default branch in three steps:

http://hg.mozilla.org/services/server-storage/rev/cd93acfa2cbe
http://hg.mozilla.org/services/server-storage/rev/c0b6bde29ea6
http://hg.mozilla.org/services/server-storage/rev/74800783777e

Comment 8

[James Bonacci [:jbonacci]]

OK. Once again, marking this Verified.