fix celery use and drop django-celery requirement



4 years ago
2 years ago


(Reporter: willkg, Assigned: rrosario)



(Whiteboard: u=dev c=codequality p= s=input.2015q2)

When we upgraded celery, we kept django-celery to reduce the work involved.

However, django-celery is deprecated and celery has a new way of integrating with Django.

This bug covers updating our use of celery to allow us to drop django-celery requirement.
We should make this a P1 and block the django 1.8 upgrade. The longer we go with django-celery, the more "in the woods and lost and could be eaten by a bear" we get.
Blocks: 1146686
Priority: -- → P1

Comment 2

4 years ago
let's start the 1.8 fun!
Assignee: nobody → rrosario


4 years ago
Depends on: 1159867


4 years ago
Depends on: 1161046
In a PR:

Also, I created bug #1161236 to ditch the "./ celeryd" launching method that this PR is creating and further fix the code to support "celery -A fjord worker -l info" launching method.
Landed in

I'll push it to stage and test it out tomorrow. Looking forward to the adventure of discovering how our -stage and -prod environments are really set up.
I pushed this to stage just now. I checked the deploy log to make sure the celery service is restarting correctly. I saw this:

[2015-05-05 06:39:14] [] running: /sbin/service celeryd-input-stage restart
[2015-05-05 06:39:27] [] finished: /sbin/service celeryd-input-stage restart (12.827s)
[] out: Restarting celery-input-stage: celery-input-stage: stopped
[] out: celery-input-stage: started
[] out: [  OK  ]
[2015-05-05 06:39:27] Finished update_celery (14.334s)

That looks good.

Further, in the admin section of the site, there's a "Celery settings and health" page which allows us to create a celery task that sends an email when it executes with details. The task was created and executed and everything looks good there, too.

I pushed it to prod. Everything looks good functionality-wise and deploy-wise.
Last Resolved: 4 years ago
Resolution: --- → FIXED
We had sporadic problems in production with "MySQL has gone away" errors.

Traceback (most recent call last):
  File "/data/www/", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/www/", line 438, in __protected_call__
    return*args, **kwargs)
  File "/data/www/", line 34, in translate_task
    instance = key_to_instance(instance_key)
  File "/data/www/", line 514, in key_to_instance
    instance = cls.objects.get(pk=int(id_))
  File "/data/www/", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/data/www/", line 351, in get
    num = len(clone)
  File "/data/www/", line 122, in __len__
  File "/data/www/", line 966, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/data/www/", line 265, in iterator
    for row in compiler.results_iter():
  File "/data/www/", line 701, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/data/www/", line 787, in execute_sql
    cursor.execute(sql, params)
  File "/data/www/", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/data/www/", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/data/www/", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/data/www/", line 129, in execute
    return self.cursor.execute(query, args)
  File "/data/www/", line 173, in execute
    self.errorhandler(self, exc, value)
  File "/data/www/", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

The problem was that DJANGO_SETTINGS_MODULE wasn't set in the environment when the celery django fixup code executes.

Fix for that and some minor cleanup in a PR:

Pushed it all to prod. I'll keep an eye on things to make sure things don't go awry.
Seems fine now. Yay!
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.