Tweak Celery options used on Heroku to match CloudAMQP recommendations

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: emorley, Assigned: emorley)

Tracking

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
In the footer of a recent email from CloudAMQP to another Heroku app (not treeherder-heroku), it suggested using these options to reduce message traffic:
`--without-gossip --without-mingle --without-heartbeat`

On their docs page some of those are mentioned, along with a few other suggestions:
https://www.cloudamqp.com/docs/celery.html

I've sent CloudAMQP an email to ask they consolidate the advice (since it was conflicting between the two sources) - which they've said they'll do (impressively they had a 4 minute response time to my email to support@).

Once that page has been updated we should consider checking we're following Celery/CloudAMQP best practice :-)
(Assignee)

Comment 1

3 years ago
They've since replied saying they've updated their docs, so we're now good to use them to tweak our settings.
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1238976
(Assignee)

Updated

3 years ago
Blocks: 1234866
No longer blocks: 1176484
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1167217
(Assignee)

Comment 4

3 years ago
This will likely need to wait until after James has finished the auto-classification work, since that branch re-enables the storing/checking of the result from each task, which negates some of the performance improvements we can make here.

As part of this we should also look at the suggestions in bug 1167217, which was duped to this.
(Assignee)

Updated

2 years ago
Blocks: 1336967
(Assignee)

Updated

2 years ago
Assignee: nobody → emorley
Created attachment 8836909 [details] [review]
[treeherder] mozilla:celery-cloudamqp > mozilla:master
(Assignee)

Comment 6

2 years ago
Comment on attachment 8836909 [details] [review]
[treeherder] mozilla:celery-cloudamqp > mozilla:master

Hopefully this will resolve bug 1336967 :-)
Attachment #8836909 - Flags: review?(james)
Attachment #8836909 - Flags: review?(james) → review+

Comment 7

2 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3929e8fe0c91e6d05c6352c96ea10e6cd2e2c097
Bug 1215102 - Re-arrange Celery preferences in settings.py

https://github.com/mozilla/treeherder/commit/3abb18b46e4c8e2ecbaa5f7d1888f0faeff2262e
Bug 1215102 - Adjust Celery settings to match CloudAMQP recommendations

CloudAMQP recommend disabling events/heartbeats/gossip/mingle messages,
in order to reduce unnecessary load on the rabbitmq server:
https://www.cloudamqp.com/docs/celery.html

For settings reference, see:
http://docs.celeryproject.org/en/3.1/configuration.html
http://docs.celeryproject.org/en/3.1/whatsnew-3.1.html#gossip-worker-worker-communication
http://docs.celeryproject.org/en/3.1/reference/celery.bin.worker.html#cmdoption-celery-worker--without-gossip

The Procfile is now becoming exceptionally verbose, however that's a
problem to consider solving in another bug, perhaps via a runner script.
(Assignee)

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.