1238806 - Convert cron jobs to run on AWS

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: General, defect)

Description

[Rob Hudson [:robhudson]]

We have a few options for how to run scheduled tasks on AWS but the recommended options from cloudops are:
1. celery beat (http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html)
2. AWS lambda (https://aws.amazon.com/lambda/)

This bug is to track the decision and work involved.

Comment 1

[Benson Wong [:mostlygeek]]

Correction: I have not recommended celery beat. 

Here's things we should evaluate: 

- lambda scheduled functions, like cron but run by AWS
  - downside is that we need to run it in lambda
  - we don't have much experience w/ lambda
  - better to use an AWS service vs running a one off EC2 box for cron
- cron box, ec2 server, just runs cron jobs.
- celery beat, not familiar with this seems to be like a cron box.

Since we already need celery to distribute tasks the lambda approach would be: 

- lambda function puts a task into celery
- celery works do the task 

This way we can reuse our architecture and do not need to worry about:

- keeping a single box up for cron
- making sure it is *only* a single box so we don't have duplicate job issues

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Currently, we have periodic tasks getting kicked off in one of two ways:

1. cron jobs
2. celerybeat scheduled tasks

Bug #1195990 covers moving the remaining cron jobs to celerybeat scheduled tasks. I think that work should continue because it makes all our scheduled tasks the same kick-off-wise plus they should be in a state they can work with whatever system we decided on for this bug.

I'll grab this bug and look into the ideas in comment #1 more.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I was working on removing the last cron jobs and either getting rid of them or converting them to celerybeat-scheduled things.

We talked a bit last week about this work after learning more about AWS infrastructure possibilities. It seems like periodic tasks will either be managed by a node running cron or some other thing that runs shell commands at specified intervals. Given that, moving everything to celerybeat seems unhelpful. Thus I'm stopping the work I was doing on this.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Lonnen suggested we look at schedule: https://pypi.python.org/pypi/schedule

Comment 5

[John Whitlock [:jwhitlock]]

We have moved to AWS (so removing this as a blocker), and continue using celerybeat. Bug 1268256 tracks using a settings-based configuration, to get away from the database-based configuration used by django-celery.

I believe there are two un-ported cron job tasks:

* Updating feeds, to get the latest Hacks posts on the front page. This could potentially be converted to a celery task
* Creating a wget mirror tarball of MDN. This is not appropriate for a celery task.

Kubernetes adds the concept of Cron Jobs, which we'll probably use more in the future:

https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

Comment 6

[John Whitlock [:jwhitlock]]

Two more unported cronjobs:

* ./manage.py delete_old_revision_ips
* ./manage.py empty_attachments_trash

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

Mass-removing myself from cc; search for 12b9dfe4-ece3-40dc-8d23-60e179f64ac1 or any reasonable part thereof, to mass-delete these notifications (and sorry!)

Comment 8

[BMO Automation]

MDN Web Docs' bug reporting has now moved to GitHub. From now on, please file content bugs at https://github.com/mdn/sprints/issues/ and platform bugs at https://github.com/mdn/kuma/issues/.