Jobvite cron job doesn't seem to be running as of Nov. 30

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: jfong, Assigned: cturra)

Tracking

Details

(Whiteboard: [triaged 20121203][push interrupt])

(Reporter)

Description

6 years ago
On http://careers.mozilla.org it appears that ./manage.py syncjobvite has stopped updating as of November 30 (last week).

Can someone check to see if there are any errors in the logs? Thanks!
(Assignee)

Comment 1

6 years ago
:ednapiranha - i just manually ran the manage.py command the cron executes and it appears to have exited in error, but is not returning any stdout to indicate what's going on. is there some way of running this more verbosely or logs i can look at?

[root@genericadm.private.phx1]# /usr/bin/flock -w 10 /var/lock/careers-prod /data/genericrhel6/src/careers.mozilla.org/lumbergh/manage.py syncjobvite

[root@genericadm.private.phx1 careers.mozilla.org]# echo $?
1
Assignee: server-ops-webops → cturra
Status: NEW → ASSIGNED
Whiteboard: [triaged 20121203][push interrupt]
(Reporter)

Comment 2

6 years ago
As a note, the cron job works on careers-dev, just not on the production server (as tested with :cturra)
(Assignee)

Comment 3

6 years ago
i have got this sorted! stage and prod `syncjobvite` processes had hung on nov 29th and as a result there were lock files still in place on the admin node running these crons. all subsequent runs were exiting in error due to the lock file being present (we use `flock` for this).

the clean up...

[root@genericadm.private.phx1 lumbergh]# ps aux | grep [c]areer
root       995  0.0  0.0   9212   616 ?        Ss   Nov29   0:00 /bin/sh -c /usr/bin/flock -w 10 /var/lock/careers-stage /data/genericrhel6-stage/src/careers.allizom.org/lumbergh/manage.py syncjobvite > /dev/null 2>&1
root      1023  0.0  0.0   9212   616 ?        Ss   Nov29   0:00 /bin/sh -c /usr/bin/flock -w 10 /var/lock/careers-prod /data/genericrhel6/src/careers.mozilla.org/lumbergh/manage.py syncjobvite > /dev/null 2>&1
root      1051  0.0  0.0   4036   368 ?        S    Nov29   0:00 /usr/bin/flock -w 10 /var/lock/careers-prod /data/genericrhel6/src/careers.mozilla.org/lumbergh/manage.py syncjobvite
root      1090  0.0  0.0 240416  1988 ?        S    Nov29   0:00 python /data/genericrhel6/src/careers.mozilla.org/lumbergh/manage.py syncjobvite
root      1097  0.0  0.0   4036   368 ?        S    Nov29   0:00 /usr/bin/flock -w 10 /var/lock/careers-stage /data/genericrhel6-stage/src/careers.allizom.org/lumbergh/manage.py syncjobvite
root      1135  0.0  0.0 240396  1992 ?        S    Nov29   0:00 python /data/genericrhel6-stage/src/careers.allizom.org/lumbergh/manage.py syncjobvite

[root@genericadm.private.phx1 lumbergh]# for p in `ps aux | grep [c]areer | awk '{print $2}'`; do kill $p; done

[root@genericadm.private.phx1 lumbergh]# ps aux | grep [c]areer | wc -l
0


... and a test run to ensure everything is running as expected once again

[root@genericadm.private.phx1 lumbergh]# /usr/bin/flock -w 10 /var/lock/careers-prod /data/genericrhel6/data/genericrhel6/src/careers.mozilla.org/lumbergh/vendor/src/django/django/db/backends/mysql/base.py:100: Warning: Incorrect string value: '\xE2\x80\xA8\xC2\xA0<...' for column 'description' at row 1
  return self.cursor.execute(query, args)
/data/genericrhel6/src/careers.mozilla.org/lumbergh/vendor/src/django/django/db/backends/mysql/base.py:100: Warning: Incorrect string value: '\xE2\x80\xA8\xE2\x80\xA8...' for column 'description' at row 1
  return self.cursor.execute(query, args)
/data/genericrhel6/src/careers.mozilla.org/lumbergh/vendor/src/django/django/db/backends/mysql/base.py:100: Warning: Incorrect string value: '\xE2\x80\xA8 <b...' for column 'description' at row 1
  return self.cursor.execute(query, args)
Synced: 82
Added: 1
Removed: 1
Removed departments: 0
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED

Comment 4

6 years ago
how often does the cron run?
(Assignee)

Comment 5

6 years ago
:cmore - it runs every 30 minutes in each (dev/stage/prod) environment.
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.