588300 - [PAMO] celeryd is not restarting

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Wil Clouser [:clouserw]]

The update script runs every 5 minutes on PAMO, but celeryd isn't being restarted with it.  The graph shows celeryd got restarted a total of twice yesterday on -app01 and not at all on -app02.

http://nm-dash01.nms.mozilla.org/ganglia/graph.php?c=amo-celery&h=pm-gearman-amo01.mozilla.org&v=37447&m=celeryd-preview%20process%20age&r=day&z=medium&jr=&js=&st=1282109442&vl=seconds&z=large

http://nm-dash01.nms.mozilla.org/ganglia/graph.php?c=amo-celery&h=pm-gearman-amo02.mozilla.org&v=206298&m=celeryd-preview%20process%20age&r=day&z=medium&jr=&js=&st=1282109653&vl=seconds&z=large

Comment 1

[Jeremy Orem [:oremj]]

The init script sucks and I'm pretty sure that is what is making this not work. I am cron restarting it after the content update every 10 minutes. Does anyone on your team have time to hack on the init script? I can post the current one somewhere.

Comment 2

[Wil Clouser [:clouserw]]

Nobody with free time in the near future :-/

Comment 3

[Jeremy Orem [:oremj]]

Alright, I should be able to look at it in in the next couple of days.

Comment 5

[Jeremy Orem [:oremj]]

There is an init script on the celery mailing list, but it appears to have the same problems as teh one we already have. I've e-mailed the list with the problems.

Comment 6

[Jeremy Orem [:oremj]]

This should be fixed now.