861447 - Deploying crontab for airmozilla

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Peter Bengtsson [:peterbe]]

Because of the work in https://bugzilla.mozilla.org/show_bug.cgi?id=848859

We now have 2 things that need to be cron run. 

1. Can you please dismantle any existing crontabs set up from doing https://bugzilla.mozilla.org/show_bug.cgi?id=857369

2. There is a script now in ./bin/crontab/gen-cron.py which generates a crontab file from ./bin/crontab/crontab.tpl (which now has the two jobs I want to run)

The setup is standard playdoh stuff so the parameters are similar to those for zamboni or kitsune (except it's `--webapp=...` rather than `--zamboni=...`)

Comment 1

[Daniel Maher [:phrawzty]]

Hello,

I'm not sure what exactly you want to have done.  On which machine(s) should "./bin/crontab/gen-cron.py" be executed, and in turn, on which machine(s) should the resulting cron jobs be implemented ?  Also, what is the schedule upon which the resulting cron jobs should be run ?

Comment 2

[Peter Bengtsson [:peterbe]]

This is the same (or very similar) solution that kitsune (SUMO) and zamboni (AMO/Marketplace) has. 

The jobs need to run on the admin node. In fact, I don't think it matters because basically all they do is connect to the database, do some "calculations" and either send out emails or change the database.

The periodicity is implied since you'll apply the whole generated crontab file. 

I don't know how else to help. Perhaps someone else in your team has experience with setting this up for other django/playdoh based projects.

Comment 3

[Daniel Maher [:phrawzty]]

Do you want this to be set up for each of dev, stage, and prod ?

Comment 4

[Peter Bengtsson [:peterbe]]

Just dev and prod. Otherwise there's a risk that stage will actually send out real tweets and real email.

Comment 5

[Daniel Maher [:phrawzty]]

Cron jobs are generally managed by Puppet.  I've integrated the relevant output from "gen-crons.py" (for dev and stage only) into Puppet, which consisted of two actual cron jobs :
* "manage.py cron send_unsent_tweets" every 5 minutes
* "manage.py cron pester_approvals" at 7 minutes past midnight daily

These will become active within the next 60 minutes.

I also pushed stage in order to ensure consistency between the admin node and the webapp.

Comment 6

[Peter Bengtsson [:peterbe]]

Awesome! Now we can use Twitter in Air Mozilla.

So, the next time I make a change to crontab.tpl it'll take effect within 60 minutes? ...on dev and prod.

Comment 7

[Daniel Maher [:phrawzty]]

(In reply to Peter Bengtsson [:peterbe] from comment #6)
> So, the next time I make a change to crontab.tpl it'll take effect within 60
> minutes?

No - as I mentioned in comment #5, cron jobs are managed by Puppet.  Puppet is, itself, managed via a protected repository, and allowing outside scripts to interact with it directly would pose an incredible security risk.  To illustrate, I ran your gen-crons.py script with the appropriate argument, captured the output, and inserted it into the appropriate cron management mechanism in Puppet manually.

That said, allowing the webapp maintainers to manage their own cronjobs is a reasonable idea, and one which I'd like to explore further (though, of course, outside of the scope of this bug).  I will bring it up with the other webops in our weekly meeting.

As an aside, it is my understanding that the Infrastructure team - who are ultimately responsible for Puppet - are working towards a public / private architecture, whereby certain elements will be manageable from the outside.  When this happens we'll be sure to let everybody know. :)

Comment 8

[Peter Bengtsson [:peterbe]]

Hmm... Perhaps this is a matter of how we deploy airmozilla. 

I talked to the SUMO people. They run the gen-crons.py script every time on deployment. 
https://github.com/mozilla/kitsune/blob/master/scripts/update/deploy.py#L52
Notice how it runs the `mv ` command as part of that task. 

Perhaps this thing about flexible control of the crontab runs into this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=816739 which is about steal some more ideas from AMO and SUMO and do the deployment the same way. 

If there's a reason why Air Mozilla can't have the same flexibility as SUMO then I'm happy with that and you can close the bug and we'll file bugs every time cron jobs need to change.

Comment 9

[Daniel Maher [:phrawzty]]

Right you are.  I didn't realise that this was something we were doing now.  Well, since there's already precedent, there's nothing stopping us from doing the same for AirMo - which, frankly stated, I'm happy to do, since it both reduces workload for me and makes things more flexible for you. :)

I've removed the noted in comment #5 from Puppet, and added the gen-crons.py step to the dev and stage deployment scripts.  It is active now.