1211651 - Use named queues for celery tasks

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Code Cleanup, defect)

Description

[Luke Crouch [:groovecoder]]

When we create celery tasks, we want to assign them to separate queues, so we can purge low-priority task queues as needed.

Comment 1

[Luke Crouch [:groovecoder]]

I updated our code [1] to split our celery tasks across 4 distinct queues (described in the PR).

It requires us to change our celery worker process commands [2] to make all the workers work on all the queues:

+worker: python2.7 manage.py celery worker --events --beat --autoreload --concurrency=4 -Q mdn_purgeable,mdn_search,mdn_emails,mdn_wiki,celery

:cyliang - can you help us merge & deploy this? You should be able to update the worker process commands immediately - they will continue to work on the default 'celery' queue where all tasks are sent now. Then, when we merge and deploy this code, celery and django will restart so new tasks will start going to separate queues, and the workers will start processing them from the separate queues too.

[1] https://github.com/mozilla/kuma/pull/3550
[2] https://github.com/mozilla/kuma/pull/3550/files#diff-5bd6b85c2d6fc987875f4bf82de2a15a

Comment 2

[C. Liang [:cyliang]]

On -stage, it looks like we have three "classes" of celery processes:

    - celerybeat: [...]python2.7 ./manage.py celerybeat --pidfile /var/run/celery/beat.pid
    - celerycam: [...]python2.7 ./manage.py celerycam --freq=2.0 --pidfile /var/run/celery/camera.pid
    - "regular" celery: [...]python2.7 ./manage.py celeryd -c 4

I've updated the invocation for celery to add the -Q flag with the list of queues listed earlier in the bug.  I've restarted celery which seems to be running correctly.  (I was able to edit a page and have it refresh correctly.)  On the rabbit server, I see new queues in the developer-stage vhost: mdn_emails, mdn_purgeable, mdn_search, and mdn_wiki. =)

I'll hold off on making similar changes to production until after you've confirmed that the new queues in -stage work as intended. =)

Comment 3

[Luke Crouch [:groovecoder]]

Thanks :cyliang. We should only need the -Q flag on the worker process I think.

I'll catch you on IRC to deploy the corresponding code with us so we can watch the stage server using the new queues successfully before going on to prod.

Comment 4

[C. Liang [:cyliang]]

Changes to celery worker flags pushed out to -prod.

Comment 5

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/8359469b835a12e0f8961c94a5a81531dea94ea6
bug 1211651 - assign tasks to certain queues

https://github.com/mozilla/kuma/commit/3834d209b481a718b859049a90b2f9155374fa9a
Merge pull request #3550 from mozilla/named-celery-queues-1211651

bug 1211651 - assign tasks to certain queues

r=robhudson

Comment 6

[Jannis Leidel [:jezdez]]

This was fixed a while ago.