1215102 - Tweak Celery options used on Heroku to match CloudAMQP recommendations

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

In the footer of a recent email from CloudAMQP to another Heroku app (not treeherder-heroku), it suggested using these options to reduce message traffic:
`--without-gossip --without-mingle --without-heartbeat`

On their docs page some of those are mentioned, along with a few other suggestions:
https://www.cloudamqp.com/docs/celery.html

I've sent CloudAMQP an email to ask they consolidate the advice (since it was conflicting between the two sources) - which they've said they'll do (impressively they had a 4 minute response time to my email to support@).

Once that page has been updated we should consider checking we're following Celery/CloudAMQP best practice :-)

Comment 1

[Ed Morley [:emorley]]

They've since replied saying they've updated their docs, so we're now good to use them to tweak our settings.

Comment 4

[Ed Morley [:emorley]]

This will likely need to wait until after James has finished the auto-classification work, since that branch re-enables the storing/checking of the result from each task, which negates some of the performance improvements we can make here.

As part of this we should also look at the suggestions in bug 1167217, which was duped to this.

Comment 5

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:celery-cloudamqp > mozilla:master

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8836909 [details] [review]
[treeherder] mozilla:celery-cloudamqp > mozilla:master

Hopefully this will resolve bug 1336967 :-)

Comment 7

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3929e8fe0c91e6d05c6352c96ea10e6cd2e2c097
Bug 1215102 - Re-arrange Celery preferences in settings.py

https://github.com/mozilla/treeherder/commit/3abb18b46e4c8e2ecbaa5f7d1888f0faeff2262e
Bug 1215102 - Adjust Celery settings to match CloudAMQP recommendations

CloudAMQP recommend disabling events/heartbeats/gossip/mingle messages,
in order to reduce unnecessary load on the rabbitmq server:
https://www.cloudamqp.com/docs/celery.html

For settings reference, see:
http://docs.celeryproject.org/en/3.1/configuration.html
http://docs.celeryproject.org/en/3.1/whatsnew-3.1.html#gossip-worker-worker-communication
http://docs.celeryproject.org/en/3.1/reference/celery.bin.worker.html#cmdoption-celery-worker--without-gossip

The Procfile is now becoming exceptionally verbose, however that's a
problem to consider solving in another bug, perhaps via a runner script.