Closed Bug 583269 Opened 14 years ago Closed 14 years ago

[amo] Rabbitmq is busted

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jbalogh, Assigned: oremj)

Details

Rabbit stopped running last night.  I don't know if this had anything to do with the ace issues.  Shyam started it back up and it seemed ok, but then it crashed (http://fox2mike.pastebin.mozilla.org/757279).  I had him rm the mnesia db, but it's still not working.  Zamboni is logging the error and moving on instead of crashing, but it would be nice to bring back the queue.

Switching to the redis backend for Celery is sounding kinda nice right now.
Oh, this is the error message: "IOError: Socket closed"
Assignee: server-ops → jeremy.orem+bugs
We forgot to set up permissions after wiping rabbit's db.  Instructions now live at https://intranet.mozilla.org/UpdateAMO#Celery.

Rabbit got busted because celery stopped processing messages.  We don't know why.  http://nm-dash01.nms.mozilla.org/ganglia/graph.php?c=amo-celery&h=pm-gearman-amo01.mozilla.org&v=12311&m=rabbit_zamboni_preview_messages_ready&r=week&z=medium&jr=&js=&st=1280525246&vl=messages&z=large
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.