1082031 - Fix bug in running periodic Celery tasks

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Code Cleanup, defect)

Description

[Jannis Leidel [:jezdez]]

What did you do?
================
See https://errormill.mozilla.org/mdn/mdn/group/172944/ for the traceback.

What happened?
==============
Some periodic Celery tasks can't seem to be run against RabbitMQ.

What should have happened?
==========================
Successful running the tasks.

Is there anything else we should know?
======================================

Comment 1

[Jannis Leidel [:jezdez]]

Seems like this is an issue deep in Celery. It was fixed in 3.x though: https://github.com/celery/celery/issues/1226

We run 2.x.

Comment 2

[Jannis Leidel [:jezdez]]

See https://github.com/mozilla/kuma/pull/2949

Comment 3

[MDN Team (:mdn-dev)]

Commits pushed to bug1082031 at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/2574ded238a3de516d7208ab5021bc1dcaf62f03
Fix bug 1082031 - Updated Celery to 3.x including its dependencies.

This also updates a bunch of dependencies:

- billiard to 3.3.0.19
- celery to 3.1.17
- django-celery to 3.1.16
- kombu to 3.0.24
- librabbitmq to 1.6.1

We also remove the previously overridden South migrations for django-celery and start to store task results in the database instead of RabbitMQ to prepare for the eventual move to Redis as the broker and the ability to see task results online.

https://github.com/mozilla/kuma/commit/57b196265bf5db5b774f1d1301024dc2abfe5251
Bug 1082031 - Add Celery camera to list of Foreman processes to enable Django admin integration.

https://github.com/mozilla/kuma/commit/6ef5fac59e7041793d31e9d1cd078fa27365e1d8
Bug 1082031 - Specify which content encoding to allow for data serialisation to prevent nagging message to show up.

https://github.com/mozilla/kuma/commit/0783e5dc75b1c307efba84fa9ccff569c1907359
Merge branch 'master' into bug1082031

Comment 4

[Jannis Leidel [:jezdez]]

Sorry, that was a mistake push of a branch to the main repo.

Comment 5

[MDN Team (:mdn-dev)]

Commits pushed to bug1082031 at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/2574ded238a3de516d7208ab5021bc1dcaf62f03
Fix bug 1082031 - Updated Celery to 3.x including its dependencies.

https://github.com/mozilla/kuma/commit/57b196265bf5db5b774f1d1301024dc2abfe5251
Bug 1082031 - Add Celery camera to list of Foreman processes to enable Django admin integration.

https://github.com/mozilla/kuma/commit/6ef5fac59e7041793d31e9d1cd078fa27365e1d8
Bug 1082031 - Specify which content encoding to allow for data serialisation to prevent nagging message to show up.

https://github.com/mozilla/kuma/commit/0783e5dc75b1c307efba84fa9ccff569c1907359
Merge branch 'master' into bug1082031

https://github.com/mozilla/kuma/commit/bbb2ab0b76eb555bc5808c8a8d714d745428d7f7
Merge branch 'master' into bug1082031

Comment 6

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/2574ded238a3de516d7208ab5021bc1dcaf62f03
Fix bug 1082031 - Updated Celery to 3.x including its dependencies.

https://github.com/mozilla/kuma/commit/57b196265bf5db5b774f1d1301024dc2abfe5251
Bug 1082031 - Add Celery camera to list of Foreman processes to enable Django admin integration.

https://github.com/mozilla/kuma/commit/6ef5fac59e7041793d31e9d1cd078fa27365e1d8
Bug 1082031 - Specify which content encoding to allow for data serialisation to prevent nagging message to show up.

https://github.com/mozilla/kuma/commit/0783e5dc75b1c307efba84fa9ccff569c1907359
Merge branch 'master' into bug1082031

https://github.com/mozilla/kuma/commit/bbb2ab0b76eb555bc5808c8a8d714d745428d7f7
Merge branch 'master' into bug1082031

https://github.com/mozilla/kuma/commit/34a7f0235ada9466534fc0cdb48511636718c2f7
Bug 1082031 - Removed librabbitmq and py-amqp again from vendor.

They are listed in compiled.txt and will be installed that way.

https://github.com/mozilla/kuma/commit/b3a4b22f07c9b13fd2c8245d45ae45231e0b1d8b
Bug 1082031 - Get rid of the wheelballs.

The wheelballs served a good purpose at the time but we've no have all the dependencies locked down in local vendor submodules and can better install them in Vagrant and Travis.

This is needed for this ticket since we want to enable us to install the new compiled components librabbitmq correctly.

https://github.com/mozilla/kuma/commit/b04508bd139fa7c133c78c405ea7d81e215c1e95
Bug 1082031 - Reduce compiler optimization to increase build speed.

https://github.com/mozilla/kuma/commit/0c4772560440859855f9287a54c12da4cdd4c727
Merge branch 'master' into bug1082031

https://github.com/mozilla/kuma/commit/19f72fabe525ff4da06ceab174f1f0f965e89daa
Bug 1082031 - Send task events so that we can track their state.

https://github.com/mozilla/kuma/commit/f66deadaa28290c10b0fea7ece8081ed0eb18e65
Bug 1082031 - Run the render document tasks as regular functions.

https://github.com/mozilla/kuma/commit/a68fb50cf9719b4a803a3438f30bac18d032e7ad
Bug 1082031 - Track when a task was started.

https://github.com/mozilla/kuma/commit/bf752f1af99fcc61ff8900e25b14868da4d2ab21
Bug 1082031 - Don't error the task just because we raise an exception.

Instead this will mark the task as failed but not errored.

https://github.com/mozilla/kuma/commit/c9d2e622ef13798fc4070d73518744cee2559ad4
Bug 1082031 - Limit render lock to an hour than 3.

https://github.com/mozilla/kuma/commit/bb35ba5a8dcfb5b025d1fe6d99f1b189ab7301a5
Bug 1082031 - Made rendering stale documents immediate by default to simplify implementation.

https://github.com/mozilla/kuma/commit/1700b8c91b6306daa56184d1c6978fbf751016a8
Merge pull request #2949 from jezdez/bug1082031

Fix bugs 1110230 and 1082031 - Updated Celery to 3.x including its dependencies.

Comment 7

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/a8d0697322671d30dbe5251eaa79f865a30a555d
Bug 1082031 - Remove amqplib since we don't need it anymore for Celery.

https://github.com/mozilla/kuma/commit/1bd05f401be5d8b90f3994e679956e330b186b96
Merge pull request #2993 from mozilla/amqplib-delete

Bug 1082031 - Remove amqplib since we don't need it anymore for Celery.