1268254 - Remove django-celery dependency

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Code Cleanup, enhancement)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
The django-celery package helps integrate Celery with Django. It was deprecated with Celery 3.1, and is no longer maintained.  It will require changes to the Kuma code base to stop using it.

Who has this problem?
=====================
Code Maintainers

How do you know that the users identified above have this problem?
==================================================================
The last change to django-celery is July 2015. It may be incompatible with future versions of Django

How are the users identified above solving this problem now?
============================================================
There is a migration path for some of django-celery's features to ones provided by the Celery package. Other features need to be abandoned or changed.  Some of these require additional infrastructure, such a new result store (Redis, etc.), or a monitoring service behind the VPN

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
* Migrate Kuma out of SCL3 and into AWS, duplicating the current infrastructure
* Add new services as needed to replace django-celery features
* Migrate the code to use the new services
* Migrate django-celery and django-celery-transactions functionality to built-in Celery functionality
* Drop the django-celery and django-celery-transactions functionality.

Is there anything else we should know?
======================================
I do not recommend making the needed infrastructure changes in SCL3, because it would require a large effort from operations and negatively impact the current stability of the SCL3 celery environment.

Comment 1

[John Whitlock [:jwhitlock]]

We've moved out of SCL3 and into AWS.

We're using a statsd-type integration to monitor the depth of celery queues with Datadog, so I think we can defer launching a monitoring service like flower until and unless we need it.

The django-celery project appears to still be lightly maintained. However, we don't need the features it provides (a database results backend or admin integration), so we should continue down the path of removing it.

We've moved to Redis for the Celery backend in production, although settings still allows the Django results backend. This didn't appear to have the desired effect, and task state is still stored in the database (https://developer.mozilla.org/admin/djcelery/taskstate/).

The Django database backend for celery beat has been relaunched as django-celery-beat, and may be a better alternative to settings-based scheduling.

django-celery-transactions was last touched in 2012. We should continue with changes to remove it.

Comment 2

[Peter Bengtsson [:peterbe]]

Making this block bug 1467518 because with Python 3 we need to upgrade kombu (otherwise, hard SyntaxError) which requires an upgrade of billiard and celery.

Your point about piece-meal moving from django-celery-transactions to django-celery-beat but I haven't done the research.

What we want is to have "post transaction tasks" ultimately and we know we don't need to store the results in the Django DB. Might be an option to attempt to kill two birds with one stone.

Comment 3

[John Whitlock [:jwhitlock]]

For future reference, here's the upgrade guide from Celery 3.1 to 4.0. There are many changes.

Comment 4

[Peter Bengtsson [:peterbe]]

Solved when we upgraded to Celery 4.

Comment 5

[Peter Bengtsson [:peterbe]]

https://github.com/mozilla/kuma/pull/5475