Closed Bug 694474 Opened 13 years ago Closed 13 years ago

[AMO] Connection refused talking to rabbit

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: clouserw, Assigned: oremj)

References

Details

Bug 694465 has a traceback we're getting in production.  It looks like rabbit (the message broker) is down.
BROKER_HOST = 'rabbit-amo'
BROKER_PORT = 5672
Just found this in the log on celery1:

[root@celery1.addons.phx1 rabbitmq]# cat rabbit@celery1.log

=ERROR REPORT==== 13-Oct-2011::17:43:36 ===
Cluster upgrade needed but other nodes are running [add_ip_to_listener,
                                                    internal_exchanges,
                                                    remove_user_scope,
                                                    topic_trie,
                                                    user_to_internal_user]
and I want [add_ip_to_listener,exchange_event_serial,exchange_scratch,gm,
            ha_mirrors,mirrored_supervisor,remove_user_scope,
            semi_durable_route,topic_trie,user_admin_to_tags]
this is being handled
Assignee: server-ops → jeremy.orem+bugs
Looks like a puppet manifest was switched around and that caused a rabbit upgrade and a version inconsistency between rabbits :-( This is fixed now.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.