socket timeouts for redis and celery

RESOLVED FIXED in 4.x (triaged)

Status

P1
normal
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: jbalogh, Assigned: jbalogh)

Tracking

4.x (triaged)

Details

(Whiteboard: [z])

(Assignee)

Description

8 years ago
Otherwise they hang our websites.
(Assignee)

Comment 1

8 years ago
Using the devious server from http://gist.github.com/527442.

Here's a branch of redis-py that will do the right thing with socket errors: http://github.com/jbalogh/redis-py/commits/connection-errors. Currently it reconnects until it gets a recursion error: http://github.com/andymccurdy/redis-py/issues/issue/62.  Then we'll have to set socket_timeout in the redis.Redis connection.
Is this something we need to patch in production?
(Assignee)

Comment 3

8 years ago
(In reply to comment #2)
> Is this something we need to patch in production?

Redis isn't critical since we have it turned off, but we should fix celery since we're using it in prod.
(Assignee)

Comment 4

8 years ago
This sets the rabbitmq timeout to 0.1 seconds: http://github.com/jbalogh/zamboni/commit/55765f4a7f

And this hijacks amqplib to not clear timeouts (wtf?): http://github.com/jbalogh/zamboni-lib/commit/a140f53d

=> 5.11.9 since redis can wait.
Target Milestone: 5.11.8 → 5.11.9
(Assignee)

Comment 5

8 years ago
The rabbit fix only works for the web side.  We don't want celeryd connections to die.
Target Milestone: 5.11.9 → 5.11.8
(Assignee)

Comment 6

8 years ago
Switched to my redis branch in http://github.com/jbalogh/zamboni-lib/commit/a97a976 and updated cache-machine to catch socket errors in http://github.com/jbalogh/django-cache-machine/commits/1eb9f2f7383bb.

I don't know what to do about celery yet.  We want persistent connections in celeryd but fail-fast sockets from the web app.
Target Milestone: 5.11.8 → 5.11.9
(Assignee)

Updated

8 years ago
Target Milestone: 5.11.9 → 4.x (triaged)
(Assignee)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.