1151514 - Stuck cron job on supportadm

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Michael Cooper [:mythmon]]

+++ This bug was initially created as a clone of Bug #1129530 +++

SUMO's cron jobs each have a unique lock so they don't end up running multiple times simultaneously.

One of SUMO's cron jobs (collect_tweets, which runs once a minute) seems to have gotten stuck. It's lock (/tmp/django_cron.sumo-prod.collect_tweets) is still in place, and the process is still running.

Can someone kill the process and delete the lock file? This cron is expected to run once a minute, and it hasn't run in a few days.

We have made changes to this cronjob so that it times out after 10 seconds, so hopefully it won't get stuck again in the future.

I get emails from these cronjobs that look like 

Cron <apache@supportadm> cd /data/support/www/support.mozilla.org/kitsune; source virtualenv/bin/activate; python -W ignore::DeprecationWarning manage.py cron collect_tweets

Script run multiple times. If this isn't true, delete `/tmp/django_cron.sumo-prod.collect_tweets`.
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit

Comment 1

[C. Liang [:cyliang]]

Whuff.  Apologies for the late response as this seems to have gotten dropped by the kanban.

I've deleted the lock file and terminated the long running process (Apr 2nd).  It looks like there have been two runs of the collect_tweets job which have (hopefully) executed correctly.  (Neither run left a log file behind.)

Comment 2

[Shyam Mani [:fox2mike]]

mythmon - Bug 1140870 is related I believe :)