587696 - socket timeouts for redis and celery

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Public Pages, defect, P1)

Description

[Jeff Balogh (:jbalogh)]

Otherwise they hang our websites.

Comment 1

[Jeff Balogh (:jbalogh)]

Using the devious server from http://gist.github.com/527442.

Here's a branch of redis-py that will do the right thing with socket errors: http://github.com/jbalogh/redis-py/commits/connection-errors. Currently it reconnects until it gets a recursion error: http://github.com/andymccurdy/redis-py/issues/issue/62.  Then we'll have to set socket_timeout in the redis.Redis connection.

Comment 2

[Wil Clouser [:clouserw]]

Is this something we need to patch in production?

Comment 3

[Jeff Balogh (:jbalogh)]

(In reply to comment #2)
> Is this something we need to patch in production?

Redis isn't critical since we have it turned off, but we should fix celery since we're using it in prod.

Comment 4

[Jeff Balogh (:jbalogh)]

This sets the rabbitmq timeout to 0.1 seconds: http://github.com/jbalogh/zamboni/commit/55765f4a7f

And this hijacks amqplib to not clear timeouts (wtf?): http://github.com/jbalogh/zamboni-lib/commit/a140f53d

=> 5.11.9 since redis can wait.

Comment 5

[Jeff Balogh (:jbalogh)]

The rabbit fix only works for the web side.  We don't want celeryd connections to die.

Comment 6

[Jeff Balogh (:jbalogh)]

Switched to my redis branch in http://github.com/jbalogh/zamboni-lib/commit/a97a976 and updated cache-machine to catch socket errors in http://github.com/jbalogh/django-cache-machine/commits/1eb9f2f7383bb.

I don't know what to do about celery yet.  We want persistent connections in celeryd but fail-fast sockets from the web app.