1448871 - Run buildhub cronjob with MIN_AGE_LAST_MODIFIED_HOURS on weekdays

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[Peter Bengtsson [:peterbe]]

We currently run the cronjob inventory cron job every day and it takes about 1.5-2h to complete. This is dangerously long and if anything goes wrong, the execution might break and fail and we have to cross our fingers it will work 24 hours later. 

As of https://github.com/mozilla-services/buildhub/issues/380, which has NOT (at the time of writing) made it to prod, you can now start the cron job with an environment variable called MIN_AGE_LAST_MODIFIED_HOURS. This significantly speeds up the job as it can quickly discard all CSV entries, after a quick date parse, that are older than X number of hours. 

I propose we start with MIN_AGE_LAST_MODIFIED_HOURS=48 and at least *once a week*, run the cronjob *without* this set. Alternatively we can set it to something like MIN_AGE_LAST_MODIFIED_HOURS=720 which means it runs for the last 30 days worth of releases in S3. 

Some more details are available in: https://github.com/mozilla-services/buildhub/issues/380

Comment 1

[Peter Bengtsson [:peterbe]]

Because I didn't release that Stage is upgraded automatically on new commits to master, I didn't release this wasn't made available yet. 

As of Release v1.2.0 this new code should be available in Stage. 
https://github.com/mozilla-services/buildhub/releases/tag/v1.2.0

Upgrades and deployments aside, we're currently running `latest-inventory-to-kinto` something like this:

 @daily latest-inventory-to-kinto

What this bug is about is changing that to something like this:

 # At 22:00 every day of them week except Sunday
 0 22 * * 1-6 MIN_AGE_LAST_MODIFIED_HOURS=48 latest-inventory-to-kinto
 # At 22:00 every Sunday
 0 22 * * 7 latest-inventory-to-kinto

(don't actually know if that's valid cron syntax!)

Comment 2

[:wezhou]

Is there a reason to run it at 22:00?

Can we run it at 17:00 like what we do now?

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to :wezhou from comment #2)
> Is there a reason to run it at 22:00?
> 
No reason at all. Just something I used to demonstrate.

> Can we run it at 17:00 like what we do now?

Go for it.

Comment 4

[Peter Bengtsson [:peterbe]]

So I guess this should be running on stage by now. It would be nice to see the outputs from the log since the main command is run with `time` as a bash prefix. You showed, but I can't remember where, that it took something like 37 minutes on the Prod server.

Comment 5

[:wezhou]

This has been deployed to -stage and -prod.