1406003 - [ops infra socorro] new rmq instance for ops stage

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Miles Crabill [:miles]]

In order to properly isolate the new ops infra Socorro pipeline, I think we should use a completely separate RMQ instance than the ones we are already paying for. That way we don't risk "crossing the streams" in any way.

There are a couple options available:
  1. pay for a new RMQ instance from CloudAMQP monthly
  2. try to spin up and run RMQ ourselves
  3. create a new queue in the existing stage RMQ and point ops infra socorro at it

As mentioned above, I strongly lean towards option 1. Option 2. is "let miles whack on this for a bit" - but I'm already fairly well tied up in other things. At least for this prototyping stage, the easiest option is 1.

Comment 1

[Miles Crabill [:miles]]

Lonnen: can we pay for and spin up a new RMQ instance with cloudaqmp per option 1. above?

Comment 2

[Lonnen :lonnen]

@miles, if you're certain you want that I can provision anytime

Comment 3

[Miles Crabill [:miles]]

I've created a new RMQ instance on CloudAMQP's free tier for testing purposes. When the time comes we can throw down for a real instance.

Comment 4

[Miles Crabill [:miles]]

CloudAMQP doesn't support metrics shipping on free-tier plans, so we will need to upgrade to get queue length, production/consumption rates, etc. in Datadog.

I will work on getting the new instance working.

Comment 5

[Miles Crabill [:miles]]

I've created a new instance and we have verified that new-stage is using it. All appears well.
We're limited on Datadog API keys (in ops land), so I'm working on figuring that out so we can get metrics for the new RMQ node sent to Datadog.

Comment 6

[Miles Crabill [:miles]]

This node is configured and we're now properly sending metrics to ops Datadog. Marking this fixed.