807023 - Datazilla Cron Problems

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Jonathan Eads ( :jeads )]

The following command has not run since Chris Turra ran it from the command line the last time I brought up this problem:

manage.py update_pushlog --repo_host=hg.mozilla.org --hours=24

The two symptoms that I'm observing:

1.) Command doesn't run regularly.  It seems to only run when run from the command line.

2.) When it's run from the command line, it runs much slower than the command runs in the development database.

Some questions to answer:

1.) Is the command currently in the process list on the machine that it runs on?
   - If so, what is it doing? (strace -p pid) How long has it been running?
   - Is there a lock file hanging around? 
       If a lock file exists it will have "update_pushlog" in the file name on the admin node in the directory where the program is run from.
   - Are there any errors in the cron log?

2.) When run from the command line with the verbosity flag set to 2 are there any errors generated?
manage.py update_pushlog --repo_host=hg.mozilla.org --hours=24 --verbosity=2



You can see the data that this command generates here:
https://datazilla.mozilla.org/refdata/pushlog/list?days_ago=12&branches=Mozilla-Inbound

If you adjust the days_ago parameter you can see that we haven't generated any new data for 11 days.

https://datazilla.mozilla.org/refdata/pushlog/list?days_ago=1&branches=Mozilla-Inbound

This command is critical for datazilla to be useful.  I have a plan "B" if this cannot be resolved but would like to try to understand what is going on in the production environment before making a significant architectural change.

Comment 1

[Chris Turra [:cturra]]

(In reply to Jonathan Eads ( :jeads ) from comment #0)
> 
> manage.py update_pushlog --repo_host=hg.mozilla.org --hours=24
> 
> The two symptoms that I'm observing:
> 
> 1.) Command doesn't run regularly.  It seems to only run when run from the
> command line.

there is a cron that should be running this every minute:

[[e-mail]3 ~]# grep -i "update_pushlog" /etc/cron.d/datazilla.mozilla.org 
* * * * * root $PYTHON_ROOT/python $DATAZILLA_HOME/manage.py update_pushlog --repo_host=hg.mozilla.org --hours=24


> 2.) When it's run from the command line, it runs much slower than the
> command runs in the development database.
> 
> Some questions to answer:
> 
> 1.) Is the command currently in the process list on the machine that it runs
> on?
>    - If so, what is it doing? (strace -p pid) How long has it been running?

no process currently running...

[[e-mail]3 ~]# ps aux | grep [u]pdate_pushlog | wc -l
0


>    - Is there a lock file hanging around? 
>        If a lock file exists it will have "update_pushlog" in the file name
> on the admin node in the directory where the program is run from.

there was one! i have removed it.

[[e-mail]3 ~]# ls -l /root/update_pushlog.lock 
-rw-r--r-- 2 root root 0 Oct 12 13:39 update_pushlog.lock


lets see if that clears things up...

Comment 2

[Chris Turra [:cturra]]

looks like the update_pushlog is running as expected again (according to `htop`). i am going to mark this bug as r/fixed. :jeads, please reopen if you see it acting strangely again.

Comment 3

[Jeff Hammel]

If this problem persists, we may want to make the script report (via e.g. email or POST to something) when it has run and e.g. what the environment is.