Closed
Bug 1144159
Opened 10 years ago
Closed 10 years ago
fix celery use and drop django-celery requirement
Categories
(Input Graveyard :: Backend, defect, P1)
Input Graveyard
Backend
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: willkg, Assigned: rrosario)
References
Details
(Whiteboard: u=dev c=codequality p= s=input.2015q2)
When we upgraded celery, we kept django-celery to reduce the work involved.
However, django-celery is deprecated and celery has a new way of integrating with Django.
This bug covers updating our use of celery to allow us to drop django-celery requirement.
Reporter | ||
Comment 1•10 years ago
|
||
We should make this a P1 and block the django 1.8 upgrade. The longer we go with django-celery, the more "in the woods and lost and could be eaten by a bear" we get.
Blocks: 1146686
Priority: -- → P1
Reporter | ||
Comment 3•10 years ago
|
||
In a PR: https://github.com/mozilla/fjord/pull/565
Also, I created bug #1161236 to ditch the "./manage.py celeryd" launching method that this PR is creating and further fix the code to support "celery -A fjord worker -l info" launching method.
Status: NEW → ASSIGNED
Reporter | ||
Comment 4•10 years ago
|
||
Landed in https://github.com/mozilla/fjord/commit/97cd6532df265eb53e8ad6833344549d78d6c355
I'll push it to stage and test it out tomorrow. Looking forward to the adventure of discovering how our -stage and -prod environments are really set up.
Reporter | ||
Comment 5•10 years ago
|
||
I pushed this to stage just now. I checked the deploy log to make sure the celery service is restarting correctly. I saw this:
[2015-05-05 06:39:14] [input-celery1.stage.webapp.phx1.mozilla.com] running: /sbin/service celeryd-input-stage restart
[2015-05-05 06:39:27] [input-celery1.stage.webapp.phx1.mozilla.com] finished: /sbin/service celeryd-input-stage restart (12.827s)
[input-celery1.stage.webapp.phx1.mozilla.com] out: Restarting celery-input-stage: celery-input-stage: stopped
[input-celery1.stage.webapp.phx1.mozilla.com] out: celery-input-stage: started
[input-celery1.stage.webapp.phx1.mozilla.com] out: [ OK ]
[2015-05-05 06:39:27] Finished update_celery (14.334s)
That looks good.
Further, in the admin section of the site, there's a "Celery settings and health" page which allows us to create a celery task that sends an email when it executes with details. The task was created and executed and everything looks good there, too.
I pushed it to prod. Everything looks good functionality-wise and deploy-wise.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 6•10 years ago
|
||
We had sporadic problems in production with "MySQL has gone away" errors.
Traceback (most recent call last):
File "/data/www/input.mozilla.org/input/vendor/src/celery/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/data/www/input.mozilla.org/input/vendor/src/celery/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/data/www/input.mozilla.org/input/fjord/translations/tasks.py", line 34, in translate_task
instance = key_to_instance(instance_key)
File "/data/www/input.mozilla.org/input/fjord/base/utils.py", line 514, in key_to_instance
instance = cls.objects.get(pk=int(id_))
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 351, in get
num = len(clone)
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 122, in __len__
self._fetch_all()
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 966, in _fetch_all
self._result_cache = list(self.iterator())
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/query.py", line 265, in iterator
for row in compiler.results_iter():
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/sql/compiler.py", line 701, in results_iter
for rows in self.execute_sql(MULTI):
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/models/sql/compiler.py", line 787, in execute_sql
cursor.execute(sql, params)
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/data/www/input.mozilla.org/input/vendor/src/django/django/db/backends/mysql/base.py", line 129, in execute
return self.cursor.execute(query, args)
File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 173, in execute
self.errorhandler(self, exc, value)
File "/data/www/input.mozilla.org/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')
The problem was that DJANGO_SETTINGS_MODULE wasn't set in the environment when the celery django fixup code executes.
Fix for that and some minor cleanup in a PR: https://github.com/mozilla/fjord/pull/566
Landed: https://github.com/mozilla/fjord/commit/4ef62e9e58ed4505efd43f7ad211a83479c29017
Reporter | ||
Comment 7•10 years ago
|
||
Pushed it all to prod. I'll keep an eye on things to make sure things don't go awry.
Reporter | ||
Comment 8•10 years ago
|
||
Seems fine now. Yay!
Updated•8 years ago
|
Product: Input → Input Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•