Rabbit stopped running last night. I don't know if this had anything to do with the ace issues. Shyam started it back up and it seemed ok, but then it crashed (http://fox2mike.pastebin.mozilla.org/757279). I had him rm the mnesia db, but it's still not working. Zamboni is logging the error and moving on instead of crashing, but it would be nice to bring back the queue. Switching to the redis backend for Celery is sounding kinda nice right now.
Oh, this is the error message: "IOError: Socket closed"
We forgot to set up permissions after wiping rabbit's db. Instructions now live at https://intranet.mozilla.org/UpdateAMO#Celery. Rabbit got busted because celery stopped processing messages. We don't know why. http://nm-dash01.nms.mozilla.org/ganglia/graph.php?c=amo-celery&h=pm-gearman-amo01.mozilla.org&v=12311&m=rabbit_zamboni_preview_messages_ready&r=week&z=medium&jr=&js=&st=1280525246&vl=messages&z=large