Open Bug 1883090 Opened 9 months ago Updated 7 months ago

[meta] move Kafka cluster off hgweb/hgssh and onto a separate cluster

Categories

(Developer Services :: Mercurial: hg.mozilla.org, task)

Product:
Developer Services
Component:
Mercurial: hg.mozilla.org
Type:
task

Tracking

(Not tracked)

People

(Reporter: sheehan, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: meta)

Move Kafka off of hgweb and onto a separate virtualized cluster. This will provide:

  • More stability to Kafka itself, since currently high load issues on hgweb can cause Kafka to fail and require manual intervention to fix.
  • Increased capacity on the existing hgweb nodes.

To complete this we will need to:

  • Stand up the virtualized cluster in MDC1.
  • Add the new cluster nodes to Ansible configs.
  • Deploy our kafka-broker Ansible role to the new nodes.
  • Grow out the cluster by adding the new nodes and test everything works as expected.
  • Shrink the cluster by removing Kafka/Zookeeper from hgweb/hgssh.

We will then have Kafka running as a separate virtualized cluster in MDC1 instead of on the hgweb nodes directly.

Depends on: 1883093

I have deployed Kafka to the new nodes and added them to the cluster. I will work on removing the old nodes from the cluster and removing Kafka/Zookeeper from hgweb/hgssh next week.

You need to log in before you can comment on or make changes to this bug.