If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

on sp-admin01: Please kill off hung cron jobs and remove lock files

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: selenamarie, Unassigned)

Tracking

Details

These processes are hung and need to be killed, and any related lockfiles removed from /tmp and /var/tmp: 

socorro  10707 10702  0 Dec06 ?        00:00:00 /bin/bash /data/socorro/application/scripts/crons/cron_update_adus.sh
socorro  11454 11453  0 Dec06 ?        00:00:00 /bin/sh -c /data/socorro/application/scripts/crons/cron_submitter.sh crash-reports.allizom.org 1000 > /dev/null 2>&1
socorro  11455 11454  0 Dec06 ?        00:00:00 /bin/sh -c /data/socorro/application/scripts/crons/cron_submitter.sh crash-reports.allizom.org 1000 > /dev/null 2>&1
socorro  11456 11455  0 Dec06 ?        00:00:01 /bin/sh -c /data/socorro/application/scripts/crons/cron_submitter.sh crash-reports.allizom.org 1000 > /dev/null 2>&1
socorro  30226 30222  0 00:05 ?        00:00:00 /bin/sh -c /data/socorro/application/scripts/crons/cron_libraries.sh > /var/log/socorro/cron_libraries.log 2>&1
socorro  30228 30226  0 00:05 ?        00:00:00 /bin/sh /data/socorro/application/scripts/crons/cron_libraries.sh
Depends on: 817718
[root@sp-admin01.phx1 ~]# kill -9 10707
[root@sp-admin01.phx1 ~]# kill -9 11454
[root@sp-admin01.phx1 ~]# kill -9 11455
[root@sp-admin01.phx1 ~]# kill -9 11456
[root@sp-admin01.phx1 ~]# kill -9 30226
[root@sp-admin01.phx1 ~]# kill -9 30228

[root@sp-admin01.phx1 tmp]# ls -lah *.lock
-rw-r--r-- 1 socorro     socorro     6 Dec  6 17:05 cron_update_adus.lock
-rw-r----- 1 sdeckelmann sdeckelmann 0 Dec  3 16:27 selena-crash-reports.allizom.org.lock


[root@sp-admin01.phx1 tmp]# rm -rf cron_update_adus.lock
[root@sp-admin01.phx1 tmp]# cd /var/tmp
[root@sp-admin01.phx1 tmp]# ls *.lock
crash-reports.allizom.org.lock
[root@sp-admin01.phx1 tmp]# rm -rf crash-reports.allizom.org.lock 
[root@sp-admin01.phx1 tmp]# 

Killed and cleaned up, Dan will review all Zeus settings for prod socorro VIPs for any possible regression and will look at the cron scripts to see what better error output we could add to get more info on what is killing these jobs
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.