1069736 - Fix possible colliding cron jobs.

Status: RESOLVED FIXED

(support.mozilla.org :: Code Quality, task, P1)

Description

[Michael Cooper [:mythmon]]

The peep+virtualenv upgrade included a few package upgrades, including an upgrade to django-cronjobs that adds cronjob locking by default. Either because of a false positive or actual colliding cron jobs, 3 jobs are spewing lots of locking warnings like:

    Script run multiple times. If this isn't true, delete `/tmp/django_cron.lock.collect_tweets`.

The 3 tasks I've seen do this are collect_tweets, enqueue_lag_monitor_task, and record_queue_size.

Based on the number and timing of the emails, I suspect there is a stale log file. This would also explain the celery report of the queue size not going down.

I suspect we need to remove the lock files, and/or disable locking on these tasks if that is safe to do so.

This is important, we should look at right away, and it is probably worth deploying on Friday for. Hopefully it should be pretty easy. 1pt and in this sprint.

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

Something fishy here. Here is one of the tasks that kicks off every 10 minutes from a cron job:

@task()
@timeit
def measure_queue_lag(queued_time):
    """A task that measures the time it was sitting in the queue.

    It saves the data to graphite via statsd.
    """
    lag = datetime.now() - queued_time
    lag = (lag.days * 3600 * 24) + lag.seconds
    statsd.gauge('rabbitmq.lag', max(lag, 0))


I don't see how this can be taking 10 minutes to collide with the next one. Maybe removing the lock files is the first thing to test.

Comment 2

[Ricky Rosario [:rrosario, :r1cky]]

Or possibly the best way forward, short term, is to go back to what we had before which is not locking tasks.

Comment 3

[Ricky Rosario [:rrosario, :r1cky]]

Oh, my bad. This isn't related to tasks.

The problem is all the cronjobs run on the same machine, that's the problem. So we need to use this setting:

If you run multiple sets of cronjobs on the same file system and need the locks to not collide, set CRONJOB_LOCK_PREFIX to something unique in your Django settings.

Comment 4

[Ricky Rosario [:rrosario, :r1cky]]

Filed Webops bug 1071036

Comment 5

[Ricky Rosario [:rrosario, :r1cky]]

yay emails stopped