1603633 - Upgrade RabbitMQ

Status: RESOLVED FIXED

(Webtools :: Pulse, enhancement, P5)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We are running a 3-year-old version of RabbitMQ. Let's upgrade.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

My concern with doing this is less that Taskcluster will malfunction in some way (although that's possible) but that other uses of pulse will malfunction.

Kim, I suspect you at least considered an upgrade at some point when your team managed this service. If you could add any context you can remember or find, that'd be great!

Comment 2

[Kim Moir [:kmoir] ET]

I have not personally considered an upgrade while on the team but I recall discussing this with the support folks from cloudampq . I can't find any notes from the conversation which is unusual for me. I did find this
https://www.cloudamqp.com/blog/2016-05-03-upgrade-clusters.html

I would suggest reaching out to them for a meeting to discuss especially since our version is so far behind and the upgrade process may be a bit different. Glob may have more information, but he is on pto right now.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks!

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

One helpful point here: we've been running staging and our dev environments off RabbitMQ 3.7.5 / Erlang 20.1 for some time with no issues.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Based on our meeting today, edunham will be taking the lead on this one..

Comment 6

[Brian Pitts]

A few thoughts for when edunham is ready to pick this up:

Based on https://www.cloudamqp.com/blog/2020-01-30-rabbitmq-erlang-upgrades.html, it appears that the upgrade process is to first upgrade to the latest version of erlang compatible with the current rabbitmq version, then upgrade to the latest version of rabbitmq compatible with that new erlang versions, then repeat the previous two steps until there are no more upgrades available. Each of those steps requires downtime.

It seems worth talking to CloudAMQP support to determine

1. exactly what sequence of erlang and rabbitmq updates we would need to take in order to get pulse to the latest version of rabbitmq 3.7.x and an estimate of how long that would take
2. What client changes may be required to go from 3.5 to 3.7

Depending on how complex the upgrade is and how risk-averse we are, it may be better to devise a plan to spin up and switch to a new 3.7 cluster rather than plan to upgrade the current 3.5 cluster.

Comment 7

[Brian Pitts]

As an aside, we may be able to improve performance on pulse today via enabling HiPE.

https://www.cloudamqp.com/blog/2014-03-31-rabbitmq-hipe.html
https://www.cloudamqp.com/blog/2018-11-26-is-hipe-production-ready.html

Comment 8

[Brian Pitts]

Attachment: pulse_versions.png

Comment 9

[Brian Pitts]

Attachment: stage_versions.png

Comment 11

[Brian Pitts]

Emily, can you share notes from any conversations you've had with taskclsuter team and with cloudampq support on this topic with me?

Comment 12

[edunham]

Brian, I forwarded you the email thread "Re: Some questions about custom SSL certificate on orange-antelope/pulse.mozilla.org" where we discussed upgrading with support.

Notes from discussions with Dustin and Coop are in https://docs.google.com/document/d/18w8GdimPBnuPb-qzr0MYP3X_HpVbdQzoKJYCuXS106g/edit#heading=h.iueazmxrxbri

Comment 13

[Brian Pitts]

Based on that correspondence Emily forwarded me, Pulse is too far behing ot upgrade in place. CloudAMQP recommends we create a new cluster and follow https://www.cloudamqp.com/docs/cluster_migration.html#seamless-migration-with-rabbitmq-queue-federation to switch to it.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Benson, here's the bug tracking the upgrade work.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

The upgrade is complete. I have shut down the old cluster, and next week I will permanently delete it.

Comment 16

[Kim Moir [:kmoir] ET]

🎉🥳🎉 Thank you Dustin for completing this work!

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

I expect I'll be getting cupcakes from CloudAMQP, too ;)

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Old cluster is deleted. Welcome to the cheaper, more upgraded future of pulse!