Closed Bug 626067 Opened 14 years ago Closed 14 years ago

[amo] Celery or Rabbit is Down

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jbalogh, Assigned: oremj)

References

Details

It looks like rabbit didn't come back after the outage last night. It's supposed to be running on pm-gearman-amo01 but the normal rabbit graphs are missing there[1]. I don't know if there are any celeryd processors running; those may need to be brought up as well. amo is dumb and does not have any monitoring for rabbit. [1]: http://nm-dash01.nms.mozilla.org/ganglia/?c=amo-celery&h=pm-gearman-amo01.mozilla.org&m=load_one&r=hour&s=descending&hc=4&mc=2
Blocks: 626053
restarted it.
I still don't see rabbitmq on the ganglia page. Do we need to page oremj?
Severity: critical → blocker
Assignee: server-ops → phong
rabbitmq-server isn't starting. oremj will be online shortly.
(In reply to comment #3) > rabbitmq-server isn't starting. oremj will be online shortly. Is there an error message?
oremj fixed it.
Assignee: phong → jeremy.orem+bugs
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.