1169696 - Use librabbitmq with Celery/Kombu for improved performance

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

The Celery docs seem to suggest using librabbitmq over the in-built py-amqp:
http://celery.readthedocs.org/en/latest/userguide/optimizing.html#librabbitmq

We should take a look :-)

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:use-librabbitmq > mozilla:master

Comment 2

[Ed Morley [:emorley]]

It looks like there's a bug in either Kombu, py-amqp or librabbitmq, since the return type from SimpleQueue.get() actually varies depending on which is used. I've filed an issue against Kombu for this:
https://github.com/celery/kombu/issues/555

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 8703603 [details] [review]
[treeherder] mozilla:use-librabbitmq > mozilla:master

The issue I've filed against Kombu may or may not be a bug they'll fix, but we're only seeing it because we're using `body` rather than the preferred `payload`, so it's easy for us to work around :-)

Comment 4

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/96ca71bb288164704b019adaf3bf945d39a07292
Bug 1169696 - Use the message payload attribute rather than raw body

`body` is the raw message content, which can vary depending on backend:
https://github.com/celery/kombu/issues/555

We should be using `payload` instead, which handles the deserialisation
for us, and isn't effected by using librabbitmq instead of py-amqp.

https://github.com/mozilla/treeherder/commit/375960c8437788526259ea2c1f6e042afec279a0
Bug 1169696 - Use librabbitmq with Celery/Kombu for improved performance

For rabbitmq transports specified using `amqp://`, Celery/Kombu will use
librabbitmq automatically if it's available, otherwise will fall back to
the pure Python (and therefore slower) amqp:
http://celery.readthedocs.org/en/latest/userguide/optimizing.html#librabbitmq

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/d1e25ed9c700e648033817e04bc73e441fe785c5
Revert "Bug 1169696 - Use librabbitmq with Celery/Kombu for improved performance"

This reverts commit 375960c8437788526259ea2c1f6e042afec279a0.

Due to:
exceptions:NotImplementedError: ssl not supported by librabbitmq, please
use pyamqp:// or stunnel

Comment 6

[Ed Morley [:emorley]]

So it turns out that Kombu prevents using TLS with librabbitmq, even though:
(a) their docs don't mention this at all
(b) librabbitmq (and rabbitmq-c that it uses) actually supports it

As such, I've reverted the librabbitmq commit (I've left the one that fixed usage of body instead of payload) and filed an issue against Kombu for allowing use of TLS with librabbitmq:
https://github.com/celery/kombu/issues/556

This didn't cause tests to fail since they don't (and can't) use TLS. It didn't fail on Heroku since publishing to Pulse isn't set up.

Comment 7

[Ed Morley [:emorley]]

This is blocked on kombu fixing:
https://github.com/celery/kombu/issues/556

Comment 8

[Ed Morley [:emorley]]

librabbitmq is not sufficiently maintained, let's stick with py-amqp.