818571 - Add in celery to solitude

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task)

Description

[Andy McKay]

I'd like solitude to have access to its own celery queue. This is completely different from any zamboni celery queue and should not be accessible from anything else.

Comment 1

[Jason Thomas [:jason]]

Seeing the following errors when trying to start celery in -dev:

/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/metlog/path.py:52: UserWarning: Module django_sha2 was already imported from /data/www/payments-dev.allizom.org/solitude/vendor/src/django-sha2/django_sha2/__init__.py, but /data/www/payments-dev.allizom.org/venv/src/django-sha2 is being added to sys.path
  import pkg_resources
/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/metlog/path.py:52: UserWarning: Module dictconfig was already imported from /data/www/payments-dev.allizom.org/solitude/vendor/src/nuggets/dictconfig.py, but /data/www/payments-dev.allizom.org/venv/src/nuggets is being added to sys.path
  import pkg_resources
/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/metlog/path.py:52: UserWarning: Module commonware was already imported from /data/www/payments-dev.allizom.org/solitude/vendor/src/commonware/commonware/__init__.py, but /data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/metlog/path.py:52: UserWarning: Module django was already imported from /data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/__init__.py, but /data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/metlog/path.py:52: UserWarning: Module cef was already imported from /data/www/payments-dev.allizom.org/solitude/vendor/lib/python/cef.py, but /data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/conf/__init__.py:110: DeprecationWarning: The SECRET_KEY setting must not be empty.
  warnings.warn("The SECRET_KEY setting must not be empty.", DeprecationWarning)
django_paranoia.reporters.log
django_paranoia.reporters.cef_
Traceback (most recent call last):
  File "/data/www/payments-dev.allizom.org/solitude/manage.py", line 27, in <module>
    manage.main()
  File "/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/funfactory/manage.py", line 143, in main
    execute_manager(current_settings)
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/core/management/__init__.py", line 459, in execute_manager
    utility.execute()
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/core/management/base.py", line 231, in execute
    self.validate()
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/core/management/base.py", line 266, in validate
    num_errors = get_validation_errors(s, app)
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/core/management/validation.py", line 30, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/db/models/loading.py", line 158, in get_app_errors
    self._populate()
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/db/models/loading.py", line 67, in _populate
    self.load_app(app_name)
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/db/models/loading.py", line 88, in load_app
    models = import_module('.models', app_name)
  File "/data/www/payments-dev.allizom.org/solitude/vendor/lib/python/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/raven/contrib/django/models.py", line 209, in <module>
    register_handlers()
  File "/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/raven/contrib/django/models.py", line 195, in register_handlers
    from raven.contrib.celery import register_signal
  File "/data/www/payments-dev.allizom.org/venv/lib/python2.6/site-packages/raven/contrib/celery/__init__.py", line 13, in <module>
    from celery.signals import after_setup_logger, task_failure
ImportError: cannot import name after_setup_logger

Comment 2

[Jason Thomas [:jason]]

rabbitmq has been configured with a seperate vhost/queue for solitude dev. Do we need payments-proxy to have access to this?

I have added the following settings to settings/local.py

# Uncomment these to activate and customize Celery:
CELERY_ALWAYS_EAGER = False  # required to activate celeryd

BROKER_HOST = 'rabbit-dev'
BROKER_PORT = 5672
BROKER_USER = 'solitude_dev'
BROKER_PASSWORD = 'solitude_dev'
BROKER_VHOST = 'solitude_dev'

CELERY_RESULT_BACKEND = 'database'
# mysql
CELERY_RESULT_DBURI = 'mysql://<removed>'

Please merge in this pull request - https://github.com/mozilla/solitude/pull/31

Comment 3

[Jason Thomas [:jason]]

jason | andym: one question, payments-proxy-dev.allizom.org, does this need access to celery broker?
andym | jason: no