952266 - Use a different AMQP backend than amqplib

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Carrot is deprecated in favor of Kombu, but let's keep carrot for the moment at least.

It claims to support pika, which is used in socorro, so it might be OK.  https://github.com/tonyg/pika

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Source inspection demonstrates that pika supports heartbeats.  It also supports reconnections.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Carrot doesn't make it easy to pass config options to Pika, though.  I'm going to look into monkeypatching it to do so.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

We should switch to a different library, but it's less urgent now.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I have a patch to use Kombu mostly ready - just needs some dusting-off.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: use-kombu.patch

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

BTW, this changes both the agent and the web components, but does not change the message format, names, etc.  So the components can be upgraded independently.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Hmm, the heartbeat stuff appears not to work - I awoke this morning to find the selfserve-agent reconnecting every second.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: use-kombu-r2.patch

There are still some keepalive issues with kombu and py-amqp, but this fixes the problems in buildapi, at least.  Amusingly, the previous version of this patched essentially opened a unique TCP connection to send heartbeats on.  Not that useful!

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

My fixes for the keepalive issues have been merged, so once we upgrade to the next released versions, we should see improved behavior there.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: use-kombu-r3.patch

I had missed some last-minute updates to test_build.py.  The production code remains the same.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8361825 [details] [diff] [review]
use-kombu-r3.patch

Review of attachment 8361825 [details] [diff] [review]:
-----------------------------------------------------------------

This is somewhat a rubberstamp, but r+. Let's not deploy it today though?

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

landed, but with automatic updating on buildapi01 already disabled (so no immediate impact)