1281112 - mxr-processor1.private.scl3.mozilla.com /projects-central.lock: 346038secs

Status: RESOLVED FIXED

(Webtools Graveyard :: MXR, defect)

Description

[matthew devney [:jedi]]

<@nagios-scl3> Tue 01:17:29 PDT [5111] mxr-processor1.private.scl3.mozilla.com:File Age - /var/lock/mxr/long is CRITICAL: CRITICAL: 1dir(s) --
                     /var/lock/mxr/long/projects-central.lock: 346038secs /var/lock/mxr/long: 2 files (http://m.mozilla.org/File+Age+-+/var/lock/mxr/long)



$ ps axo pid,etime,command | grep projects-central
15678  4-00:09:58 /bin/sh -c echo -n "projects-central: Starting " && date && nice -n 19 ./update-full-onetree.sh -cron projects-central; echo -n "projects-central: Ending " && date
15682  4-00:09:58 /bin/sh ./update-full-onetree.sh -cron projects-central
25513       00:01 grep projects-central
30522    12:32:10 /bin/sh ./update-full-onetree.sh -cron projects-central
30523    12:32:10 perl update-search.pl -cron projects-central
30526    12:32:10 sh -c ( glimpseindex -o -n -f -B -M 128 -H . /data/mxr-data/projects-central/projects 2>&1) >> /data/mxr-data/projects-central/glimpseindex.log
30527    12:32:10 glimpseindex -o -n -f -B -M 128 -H . /data/mxr-data/projects-central/projects
[[e-mail]3 ~]$

Comment 1

[matthew devney [:jedi]]

killed youngest pid (30526) as instructed by runbook.
After 5 minutes killed next 2 (30523, 30522)
Another 5 minutes killed 30527.
Ran cron script: 
$ sudo /data/www/mxr.mozilla.org/update-full-onetree.sh projects-central

...and it recovered.

 <@nagios-scl3> Tue 01:32:27 PDT [5114] mxr-processor1.private.scl3.mozilla.com:File Age - /var/lock/mxr/long is OK: OK: 1dir(s) -- /var/lock/mxr/long: 1 files
                     (http://m.mozilla.org/File+Age+-+/var/lock/mxr/long)

Comment 2

[matthew devney [:jedi]]

killed youngest pid (30526) as instructed by runbook.
After 5 minutes killed next 2 (30523, 30522)
Another 5 minutes killed 30527.
Ran cron script: 
$ sudo /data/www/mxr.mozilla.org/update-full-onetree.sh projects-central

...and it recovered.

 <@nagios-scl3> Tue 01:32:27 PDT [5114] mxr-processor1.private.scl3.mozilla.com:File Age - /var/lock/mxr/long is OK: OK: 1dir(s) -- /var/lock/mxr/long: 1 files
                     (http://m.mozilla.org/File+Age+-+/var/lock/mxr/long)

Comment 3

[Peter Radcliffe [:pir]]

:fubar, are these doing something useful given current mxr state?

Comment 4

[Kendall Libby [:fubar] (he/him)]

(In reply to Peter Radcliffe [:pir] from comment #3)
> :fubar, are these doing something useful given current mxr state?

Nope! 

I've removed the cron jobs and their monitoring. system-level monitoring left in place, as they're still up for archeology.