In the footer of a recent email from CloudAMQP to another Heroku app (not treeherder-heroku), it suggested using these options to reduce message traffic: `--without-gossip --without-mingle --without-heartbeat` On their docs page some of those are mentioned, along with a few other suggestions: https://www.cloudamqp.com/docs/celery.html I've sent CloudAMQP an email to ask they consolidate the advice (since it was conflicting between the two sources) - which they've said they'll do (impressively they had a 4 minute response time to my email to support@). Once that page has been updated we should consider checking we're following Celery/CloudAMQP best practice :-)
They've since replied saying they've updated their docs, so we're now good to use them to tweak our settings.
This will likely need to wait until after James has finished the auto-classification work, since that branch re-enables the storing/checking of the result from each task, which negates some of the performance improvements we can make here. As part of this we should also look at the suggestions in bug 1167217, which was duped to this.
Created attachment 8836909 [details] [review] [treeherder] mozilla:celery-cloudamqp > mozilla:master
Comment on attachment 8836909 [details] [review] [treeherder] mozilla:celery-cloudamqp > mozilla:master Hopefully this will resolve bug 1336967 :-)
Attachment #8836909 - Flags: review?(james)
Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/3929e8fe0c91e6d05c6352c96ea10e6cd2e2c097 Bug 1215102 - Re-arrange Celery preferences in settings.py https://github.com/mozilla/treeherder/commit/3abb18b46e4c8e2ecbaa5f7d1888f0faeff2262e Bug 1215102 - Adjust Celery settings to match CloudAMQP recommendations CloudAMQP recommend disabling events/heartbeats/gossip/mingle messages, in order to reduce unnecessary load on the rabbitmq server: https://www.cloudamqp.com/docs/celery.html For settings reference, see: http://docs.celeryproject.org/en/3.1/configuration.html http://docs.celeryproject.org/en/3.1/whatsnew-3.1.html#gossip-worker-worker-communication http://docs.celeryproject.org/en/3.1/reference/celery.bin.worker.html#cmdoption-celery-worker--without-gossip The Procfile is now becoming exceptionally verbose, however that's a problem to consider solving in another bug, perhaps via a runner script.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.