589260 - Rabbit monitoring broke /monitor/

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Code Quality, defect)

Description

[Wil Clouser [:clouserw]]

Traceback (most recent call last):

 File "/data/amo_python/www/preview/zamboni/vendor/src/django/django/core/handlers/base.py", line 100, in get_response
   response = callback(request, *callback_args, **callback_kwargs)

 File "/data/amo_python/www/preview/zamboni/vendor/src/django/django/views/decorators/cache.py", line 69, in _wrapped_view_func
   response = view_func(request, *args, **kwargs)

 File "/data/amo_python/www/preview/zamboni/vendor/src/django/django/views/decorators/cache.py", line 69, in _wrapped_view_func
   response = view_func(request, *args, **kwargs)

 File "/data/amo_python/www/preview/zamboni/apps/amo/views.py", line 126, in monitor
   pong = celery.task.ping()

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/task/__init__.py", line 80, in ping
   return PingTask.apply_async().get()

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/result.py", line 53, in get
   return self.wait(timeout=timeout)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/result.py", line 68, in wait
   return self.backend.wait_for(self.task_id, timeout=timeout)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/backends/base.py", line 86, in wait_for
   status = self.get_status(task_id)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/backends/base.py", line 157, in get_status
   return self.get_task_meta(task_id)["status"]

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/backends/base.py", line 175, in get_task_meta
   meta = self._get_task_meta_for(task_id)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/backends/amqp.py", line 94, in _get_task_meta_for
   consumer = self._consumer_for_task_id(task_id, connection)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/celery/celery/backends/amqp.py", line 63, in _consumer_for_task_id
   routing_key=routing_key)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/carrot/carrot/messaging.py", line 230, in __init__
   self.declare()

 File "/data/amo_python/www/preview/zamboni/vendor/packages/carrot/carrot/messaging.py", line 272, in declare
   auto_delete=self.auto_delete)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/carrot/carrot/backends/pyamqplib.py", line 243, in exchange_declare
   auto_delete=auto_delete)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/amqplib/amqplib/client_0_8/channel.py", line 843, in exchange_declare
   (40, 11),    # Channel.exchange_declare_ok

 File "/data/amo_python/www/preview/zamboni/vendor/packages/amqplib/amqplib/client_0_8/abstract_channel.py", line 89, in wait
   self.channel_id, allowed_methods)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/amqplib/amqplib/client_0_8/connection.py", line 218, in _wait_method
   self.wait()

 File "/data/amo_python/www/preview/zamboni/vendor/packages/amqplib/amqplib/client_0_8/abstract_channel.py", line 105, in wait
   return amqp_method(self, args)

 File "/data/amo_python/www/preview/zamboni/vendor/packages/amqplib/amqplib/client_0_8/connection.py", line 367, in _close
   raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id))

AMQPConnectionException: (530, u"NOT_ALLOWED - cannot redeclare exchange 'celeryresults' in vhost 'zamboni_preview' with different type, durable or autodelete value", (40, 10), 'Channel.exchange_declare')

Comment 1

[Jeff Balogh (:jbalogh)]

I don't know what this means.

Comment 2

[Jeff Balogh (:jbalogh)]

This is what I see in vendor:

        backend.exchange_declare(exchange=self.exchange,
                                 type="direct",
                                 durable=True,
                                 auto_delete=False)

oremj: what does rabbitmqctl say about these values for the "celeryresults" exchange?

Comment 3

[Jeremy Orem [:oremj]]

[root@pm-gearman-amo01 ~]# rabbitmqctl list_exchanges -p zamboni_preview name type durable auto_delete arguments

Listing exchanges ...
amq.topic	topic	true	false	[]
celery	direct	true	false	[]
celeryevent	direct	true	false	[]
amq.fanout	fanout	true	false	[]
celeryctl	fanout	true	false	[]
amq.headers	headers	true	false	[]
amq.direct	direct	true	false	[]
celeryresults	direct	true	false	[]
amq.match	headers	true	false	[]
	direct	true	false	[]
...done.

Comment 4

[Jeff Balogh (:jbalogh)]

oops, should have been looking at this one:

        return Consumer(connection, queue=routing_key,
                        exchange=self.exchange,
                        exchange_type="direct",
                        no_ack=False, auto_ack=False,
                        auto_delete=True,
                        routing_key=routing_key)

Comment 5

[Wil Clouser [:clouserw]]

With rabbit not running I get a traceback that should be caught:


Traceback (most recent call last):

  File "/home/clouserw/.virtualenvs/zamboni/src/django/django/core/servers/basehttp.py", line 280, in run
    self.result = application(self.environ, self.start_response)

  File "/home/clouserw/.virtualenvs/zamboni/src/django/django/core/servers/basehttp.py", line 674, in __call__
    return self.application(environ, start_response)

  File "/home/clouserw/.virtualenvs/zamboni/src/django/django/core/handlers/wsgi.py", line 241, in __call__
    response = self.get_response(request)

  File "/home/clouserw/.virtualenvs/zamboni/src/django/django/core/handlers/base.py", line 141, in get_response
    return self.handle_uncaught_exception(request, resolver, sys.exc_info())

  File "/home/clouserw/.virtualenvs/zamboni/src/django/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/home/clouserw/.virtualenvs/zamboni/src/django/django/views/decorators/cache.py", line 69, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)

  File "/home/clouserw/.virtualenvs/zamboni/src/django/django/views/decorators/cache.py", line 69, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)

  File "/home/clouserw/dev/zamboni/apps/amo/views.py", line 126, in monitor
    pong = celery.task.ping()

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/celery/task/__init__.py", line 80, in ping
    return PingTask.apply_async().get()

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/celery/task/base.py", line 298, in apply_async
    return apply_async(self, args, kwargs, **options)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/celery/messaging.py", line 175, in _inner
    return fun(*args, **kwargs)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/celery/execute/__init__.py", line 83, in apply_async
    exchange_type=exchange_type)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/celery/task/base.py", line 249, in get_publisher
    routing_key=self.routing_key)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/celery/messaging.py", line 44, in __init__
    consumers = get_consumer_set(self.connection)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/celery/messaging.py", line 195, in get_consumer_set
    backend=cset.backend, **queue_options)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/messaging.py", line 230, in __init__
    self.declare()

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/messaging.py", line 267, in declare
    warn_if_exists=self.warn_if_exists)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/backends/pyamqplib.py", line 233, in queue_declare
    return self.channel.queue_declare(queue=queue,

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/backends/pyamqplib.py", line 179, in channel
    self._channel_ref = weakref.ref(self.connection.get_channel())

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/connection.py", line 150, in get_channel
    return self.connection.channel()

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/connection.py", line 120, in connection
    self._connection = self._establish_connection()

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/connection.py", line 133, in _establish_connection
    return self.create_backend().establish_connection()

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/carrot/backends/pyamqplib.py", line 195, in establish_connection
    connect_timeout=conninfo.connect_timeout)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/amqplib/client_0_8/connection.py", line 125, in __init__
    self.transport = create_transport(host, connect_timeout, ssl)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 220, in create_transport
    return TCPTransport(host, connect_timeout)

  File "/home/clouserw/.virtualenvs/zamboni/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 58, in __init__
    self.sock.connect((host, port))

  File "<string>", line 1, in connect

error: [Errno 111] Connection refused

Comment 6

[Wil Clouser [:clouserw]]

Also, if celeryd is not running or is processing very slowly the code appears to wait indefinitely for a response, locking up the entire page.

Comment 7

[Jeff Balogh (:jbalogh)]

(In reply to comment #6)
> Also, if celeryd is not running or is processing very slowly the code appears
> to wait indefinitely for a response, locking up the entire page.

Yeah, that's our big problem with rabbit at the moment.

Comment 8

[Jeff Balogh (:jbalogh)]

Reverted in http://github.com/jbalogh/zamboni/commit/73dac3c, reopening bug 584485.