561459 - [amo] Install RabbitMQ

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Jeff Balogh (:jbalogh)]

Please install the RabbitMQ server for AMO.  You can get source code or rpms from http://www.rabbitmq.com/server.html.  It requires Erlang.

We'll want at least one machine acting as the message broker (running rabbitmq).  A backup is probably a good idea, but I'm not sure how that works yet.  This is Erlang, so it's not supposed to crash anyways.

Once we have the queue running, we'll be using http://github.com/ask/celery/ to manage workers.  `./manage.py celeryd` will connect to rabbitmq and wait for tasks.  One celeryd creates (# of cpu cores) workers.  We can start celeryd on many machines.

celeryd will run from a checkout of zamboni.  It will need to connect to rabbitmq, memcached, and the database.

To get set up on my Mac, I did this:

brew install rabbitmq
rabbitmq-server 
rabbitmqctl add_user jbalogh foo
rabbitmqctl add_vhost zamboni
rabbitmqctl set_permissions -p zamboni jbalogh '' '.*' '.*'

The commands are from http://ask.github.com/celery/getting-started/broker-installation.html.  This is the rabbitmq admin guide: http://www.rabbitmq.com/admin-guide.html.  The permissions are (configure, write, read).

The user, password, vhost, and other connection information will be configurable through zamboni's settings_local.py.

Thanks!

Comment 1

[Jeff Balogh (:jbalogh)]

Munin plugins for rabbitmq: http://github.com/ask/rabbitmq-munin/

Comment 2

[Jeff Balogh (:jbalogh)]

I added our first queue task in http://github.com/jbalogh/zamboni/commit/6f7d29582b86

These are the celery settings you may want to change in settings_local.py: http://github.com/jbalogh/zamboni/commit/6f7d29582b867755fd91996f64707be548416bbf#L4R392

To start a task:
rabbitmq-server &
./manage.py celeryd &
./manage.py cron update_addon_appsupport

rabbitmy-server and celeryd should be daemonized to get this all going properly.

The cronjob adds a bunch of tasks, and they send debug output to the logs.  Right now the appsupport table should be empty, and the tasks will slowly fill it up, so that's another way to check if this is going.

celeryd will have to be restarted when we update zamboni to pick up new/edited tasks.  I'll look into the best way to do that.

Comment 3

[Jeff Balogh (:jbalogh)]

Oh, and http://github.com/jbalogh/zamboni/commit/c819111f2dc added celery to the requirements file.

Comment 4

[Jeff Balogh (:jbalogh)]

Celery has init.d[1] and supervisord[2] scripts in its contrib module.

[1]: http://github.com/ask/celery/tree/master/contrib/debian/init.d/
[2]: http://github.com/ask/celery/tree/master/contrib/supervisord/

Comment 5

[Jeremy Orem [:oremj]]

This is pretty much done.  Dave Dash is working on a redhat init script for celeryd.

Comment 6

[Jeremy Orem [:oremj]]

I've installed the init script and it seems to be working well.