If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Status

Infrastructure & Operations
WebOps: Other
--
critical
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: jbalogh, Assigned: oremj)

Tracking

Details

(Reporter)

Description

7 years ago
Rabbit stopped running last night.  I don't know if this had anything to do with the ace issues.  Shyam started it back up and it seemed ok, but then it crashed (http://fox2mike.pastebin.mozilla.org/757279).  I had him rm the mnesia db, but it's still not working.  Zamboni is logging the error and moving on instead of crashing, but it would be nice to bring back the queue.

Switching to the redis backend for Celery is sounding kinda nice right now.
(Reporter)

Comment 1

7 years ago
Oh, this is the error message: "IOError: Socket closed"
Assignee: server-ops → jeremy.orem+bugs
(Reporter)

Comment 2

7 years ago
We forgot to set up permissions after wiping rabbit's db.  Instructions now live at https://intranet.mozilla.org/UpdateAMO#Celery.

Rabbit got busted because celery stopped processing messages.  We don't know why.  http://nm-dash01.nms.mozilla.org/ganglia/graph.php?c=amo-celery&h=pm-gearman-amo01.mozilla.org&v=12311&m=rabbit_zamboni_preview_messages_ready&r=week&z=medium&jr=&js=&st=1280525246&vl=messages&z=large
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.