1447412 - set up stage submitter in -new-prod infra (post migration)

Status: RESOLVED DUPLICATE of bug 1430937

(Socorro :: Infra, task, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We currently have a stage submitter node in webeng that listens to the socorro.stagesubmitter RabbitMQ queue, pulls that data from S3, and then submits those crashes to the collectors in the -stage, -new-stage, and -new-prod environments.

We were thinking we'd rewrite the stage submitter as an AWS lambda job (bug #1430937), but that never happened (see bug for details).

We need to build a stage submitter node in the -new-prod environment that listens to the socorro.stagesubmitter RabbitMQ queue in -new-prod, pulls the crash data from the S3 crash bucket associated with -new-prod, and submits those crashes to the -new-stage environment.

This bug covers that work.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

We talked about this in the meeting and decided to defer this kind of work until after the migration.

We'll keep the -prod stage submitter running during the migration. -new-prod will queue crashes in both socorro.normal and socorro.stagesubmitter in the -new-prod RabbitMQ queue. Everything will go swimmingly.

At some point after the migration has been declared a success, we'll change the configuration in -prod for the -prod stage submitter to pull from the -new-prod RabbitMQ queue and the -new-prod S3 crash bucket and submit to -new-stage. Then we'll run on that until either we rewrite the stage submitter as a lambda job, we create a node in -new-prod, or some third option.

I'll do the stage submitter work, so I'll toss this on my plate. Also, I'm going to change this to block on the migration bug.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm pretty sure bug #1450683 is from the stage submitter starting up, submitting crashes, but exiting before the thread charged with ack'ing the crashes to RabbitMQ has a chance to ack them. Thus if there are a small number of crashes in the queue, they stay in the queue and never get removed.

I think that's a fundamental flaw with that app.

We could probably fix it with the existing app, but I'm inclined to not bother and instead just rewrite the stage submitter. That's covered in bug #1430937.

Given that, I'm going to change my mind about everything again and DUPE this bug to that one.