1429526 - [ops infra socorro] get adi data into -prod-new

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

When we set up -stage-new, we created a RawADIMoverCronApp crontabber job that copies data from the -stage database into the -stage-new database. That way the hive crontabber job in scl3 can continue to push data to the -prod and -stage databases and we don't have to change the script at all.

We need to do a similar thing with -prod-new. We need to run the RawADIMoverCronApp in -prod-new and have it pull from a database and populate the -prod-new db.

This bug covers figuring out exactly how that should work and setting it up.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

We could pull from the -stage db.

Pros:

1. Fewer things can access the -prod db.

Cons:

1. Will has concerns that -prod and -stage aren't the same in non-trivial ways that might affect the adi data in hand-wavey ways. So pulling from -stage might be bad.


If we pull from -stage or -prod, we still have to keep both the -stage and -prod databases around because otherwise the hive script will likely fail.

I think my vote is we have -prod-new pull from -prod and continue to pray harder and more fervently at the GetRidOfADI altar.

Comment 2

[Miles Crabill [:miles]]

I think you're right that having -prod-new pull from -prod is probably the most reasonable approach.

cc'ing :g-k for his input. The gist is that we will have to assign an EIP to the crontabber in -new-prod and whitelist that IP to the -prod socorro database. This EIP should at no point be released, and will live in the crontabber's resources cloudformation stack. We're already doing this in -new-stage (connecting to -stage's RDS instance).

Comment 3

[Greg G]

-prod to -prod-new makes sense. What's the impact of the -new-prod crontabber EIP leaking?

Comment 4

[Miles Crabill [:miles]]

If the EIP were to be released and re-associated by an instance out of our control, it could connect the -prod database server. It wouldn't have login credentials, just the ability to connect. We would have some indication within a day as the crontabber instance wouldn't be able to connect to the -prod database for the ADI job to run.

We would have to either manually release the EIP or delete the crontabber resources cloudformation stack in order to release the EIP (which, if we had not already deleted the app stack, would cause an error because of cloudformation dependencies). These are both highly unlikely scenarios.

Comment 5

[Greg G]

OK -prod to -prod-new is fine by me.

Comment 6

[Miles Crabill [:miles]]

This is configured and, I believe, working. It might be good to verify more directly.