1144159 - fix celery use and drop django-celery requirement

Status: RESOLVED FIXED

(Input Graveyard :: Backend, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

When we upgraded celery, we kept django-celery to reduce the work involved.

However, django-celery is deprecated and celery has a new way of integrating with Django.

This bug covers updating our use of celery to allow us to drop django-celery requirement.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

We should make this a P1 and block the django 1.8 upgrade. The longer we go with django-celery, the more "in the woods and lost and could be eaten by a bear" we get.

Comment 2

[Ricky Rosario [:rrosario, :r1cky]]

let's start the 1.8 fun!

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

In a PR: https://github.com/mozilla/fjord/pull/565

Also, I created bug #1161236 to ditch the "./manage.py celeryd" launching method that this PR is creating and further fix the code to support "celery -A fjord worker -l info" launching method.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in https://github.com/mozilla/fjord/commit/97cd6532df265eb53e8ad6833344549d78d6c355

I'll push it to stage and test it out tomorrow. Looking forward to the adventure of discovering how our -stage and -prod environments are really set up.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I pushed this to stage just now. I checked the deploy log to make sure the celery service is restarting correctly. I saw this:

[2015-05-05 06:39:14] [input-celery1.stage.webapp.phx1.mozilla.com] running: /sbin/service celeryd-input-stage restart
[2015-05-05 06:39:27] [input-celery1.stage.webapp.phx1.mozilla.com] finished: /sbin/service celeryd-input-stage restart (12.827s)
[input-celery1.stage.webapp.phx1.mozilla.com] out: Restarting celery-input-stage: celery-input-stage: stopped
[input-celery1.stage.webapp.phx1.mozilla.com] out: celery-input-stage: started
[input-celery1.stage.webapp.phx1.mozilla.com] out: [  OK  ]
[2015-05-05 06:39:27] Finished update_celery (14.334s)

That looks good.

Further, in the admin section of the site, there's a "Celery settings and health" page which allows us to create a celery task that sends an email when it executes with details. The task was created and executed and everything looks good there, too.

I pushed it to prod. Everything looks good functionality-wise and deploy-wise.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

We had sporadic problems in production with "MySQL has gone away" errors.

Traceback (most recent call last):
  File "/data/www/input.mozilla.org/input/vendor/src/celery/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/celery/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/fjord/translations/tasks.py", line 34, in translate_task
    instance = key_to_instance(instance_key)
  File "/data/www/input.mozilla.org/input/fjord/base/utils.py", line 514, in key_to_instance
    instance = cls.objects.get(pk=int(id_))
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 351, in get
    num = len(clone)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 122, in __len__
    self._fetch_all()
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 966, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 265, in iterator
    for row in compiler.results_iter():
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/sql/compiler.py", line 701, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/sql/compiler.py", line 787, in execute_sql
    cursor.execute(sql, params)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/mysql/base.py", line 129, in execute
    return self.cursor.execute(query, args)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)
  File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

The problem was that DJANGO_SETTINGS_MODULE wasn't set in the environment when the celery django fixup code executes.

Fix for that and some minor cleanup in a PR: https://github.com/mozilla/fjord/pull/566

Landed: https://github.com/mozilla/fjord/commit/4ef62e9e58ed4505efd43f7ad211a83479c29017

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pushed it all to prod. I'll keep an eye on things to make sure things don't go awry.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Seems fine now. Yay!