1079701 - Perform a |service supervisord restart| on production deploy

Status: RESOLVED DUPLICATE of bug 1072437

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

21:14	mdoglio	the celery worker on production are only listening to the default queue
21:14	mdoglio	that's one problem for sure
21:15	mdoglio	fubar: can you please check that the celery worker on the admin node is currently using https://github.com/mozilla/treehe...ob/master/bin/run_celery_worker to start?
21:16	fubar	command=/data/treeherder/src/treeherder.mozilla.org/treeherder-service/bin/run_celery_worker
21:17	mdoglio	fubar: I would try with a service supervisord restart then

Mauro, should we make update.py do a |service supervisord restart| too then?

Comment 1

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #0)
> Mauro, should we make update.py do a |service supervisord restart| too then?

That is to pick up future changes like bug 1076774 and bug 1076778 without requiring manual intervention.

Comment 2

[Mauro Doglio [:mdoglio]]

Restarting supervisord was our first idea to restart the celery workers, but for some reason it was sometimes creating zombie processes. I still think it's a good idea to broadcast a restart signal using celery itself, I just need to understand why sometimes that doesn't work. In order to do that, I need better monitoring of the workers, a good solution for that could [1]. Maybe fubar can help us to deploy it on the admin node and make it accessible only under vpn?

[1]http://celery.readthedocs.org/en/latest/userguide/monitoring.html#flower-real-time-celery-web-monitor

Comment 3

[Ed Morley [:emorley]]

(In reply to Mauro Doglio [:mdoglio] from comment #2)
> Restarting supervisord was our first idea to restart the celery workers, but
> for some reason it was sometimes creating zombie processes. I still think
> it's a good idea to broadcast a restart signal using celery itself, I just
> need to understand why sometimes that doesn't work. In order to do that, I
> need better monitoring of the workers, a good solution for that could [1].
> Maybe fubar can help us to deploy it on the admin node and make it
> accessible only under vpn?
> 
> [1]http://celery.readthedocs.org/en/latest/userguide/monitoring.html#flower-
> real-time-celery-web-monitor

Bug 1072437 is filed for this - let's move discussion over there :-)