806396 - Rabbit having problems on -stage?

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task)

Description

[Wil Clouser [:clouserw]]

We're getting this email:

[amo-developers] Cron <apache@addonsadm> cd /data/addons-stage/src/addons.allizom.org/zamboni; /data/addons-stage/src/addons.allizom.org/venv/bin/python -W ignore::DeprecationWarning manage.py cron update_addons_current_version

Traceback (most recent call last):
  File "manage.py", line 122, in <module>
    execute_manager(settings)
  File "/data/addons-stage/src/addons.allizom.org/venv/lib/python2.6/site-packages/django/core/management/__init__.py", line 459, in execute_manager
    utility.execute()
  File "/data/addons-stage/src/addons.allizom.org/venv/lib/python2.6/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/addons-stage/src/addons.allizom.org/venv/lib/python2.6/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/addons-stage/src/addons.allizom.org/venv/lib/python2.6/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/data/addons-stage/src/addons.allizom.org/venv/lib/python2.6/site-packages/cronjobs/management/commands/cron.py", line 64, in handle
    registered[script](*args)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/apps/addons/cron.py", line 70, in update_addons_current_version
    TaskSet(ts).apply_async()
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/celery/task/sets.py", line 132, in apply_async
    pub = publisher or self.Publisher(connection=conn)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/celery/app/amqp.py", line 328, in TaskPublisher
    return TaskPublisher(*args, **self.app.merge(defaults, kwargs))
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/celery/app/amqp.py", line 158, in __init__
    super(TaskPublisher, self).__init__(*args, **kwargs)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/kombu/compat.py", line 61, in __init__
    super(Publisher, self).__init__(connection, self.exchange, **kwargs)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/kombu/messaging.py", line 69, in __init__
    channel = channel.default_channel
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/kombu/connection.py", line 597, in default_channel
    self.connection
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/kombu/connection.py", line 590, in connection
    self._connection = self._establish_connection()
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/kombu/connection.py", line 550, in _establish_connection
    conn = self.transport.establish_connection()
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/kombu/transport/amqplib.py", line 277, in establish_connection
    connect_timeout=conninfo.connect_timeout)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/kombu/transport/amqplib.py", line 87, in __init__
    super(Connection, self).__init__(*args, **kwargs)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/amqplib/client_0_8/connection.py", line 129, in __init__
    self.transport = create_transport(host, connect_timeout, ssl)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/amqplib/client_0_8/transport.py", line 281, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/data/addons-stage/src/addons.allizom.org/zamboni/vendor/lib/python/amqplib/client_0_8/transport.py", line 85, in __init__
    raise socket.error, msg
socket.error: [Errno 111] Connection refused

Comment 1

[Jason Thomas [:jason]]

Rabbitmq rpm was upgraded recently and the cluster failed to start due to a stray lock file. This is now fixed.