1194142 - Figure out the correct value for Celery's BROKER_POOL_LIMIT on Heroku

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

We currently set BROKER_POOL_LIMIT to 1 on Heroku:
https://github.com/mozilla/treeherder/blob/1c00ccfcc74275f3ff7366c6150614bb065a8dc2/treeherder/settings/base.py#L434

However I think we should not.

The docs mention using 1, but that's more if you're using the free plan:
https://devcenter.heroku.com/articles/cloudamqp#celery

According to http://docs.celeryproject.org/en/latest/configuration.html#broker-pool-limit :
"""
The maximum number of connections that can be open in the connection pool.
...
Default (since 2.5) is to use a pool of 10 connections.
"""

We're on the cloudamqp 'Big Bunny' plan, which supports up to 5000 connections:
https://addons.heroku.com/cloudamqp?#bunny

Comment 1

[Ed Morley [:emorley]]

Attachment: Use the default Celery BROKER_POOL_LIMIT on Heroku

Comment 2

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/97a67048b5416d02c71041e317a4a6b981e45ac2
Bug 1194142 - Use the default Celery BROKER_POOL_LIMIT on Heroku

The default value of 10 is currently used by stage/prod, however on
Heroku it was set to 1, presumably since the guide here suggests it:
https://devcenter.heroku.com/articles/cloudamqp#celery

However, we're not using the free CloudAMQP plan mentioned by the guide,
but instead use their 'Big Bunny' plan, which supports up to 5000
concurrent connections.