1309395 - Treeherder Terraform cleanup post Heroku migration

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

No rush, but at some point we can:
* Adjust the treeherder-{stage,prod} RDS instances to use the default security group/subnets
* Remove the unused treeherder admin EC2 instance
* Remove the now unused parameter groups {th-import, th-replication}
* Remove the treeherder-dbgrp subnet group
* Remove the treeherder-vpc VPC
* Fix the various boolean values that were listed as strings (eg `multi_az = "True"` 
* Make the tags consistent across prototype/stage/prod

Comment 1

[Kendall Libby [:fubar] (he/him)]

I'm actually inclined to leave the security groups, subnet group and VPC as they are. Despite the DBs being public (a pox on heroku), it keeps things that don't need to talk segregated. It doesn't hurt anything, and creates work for, imo, no gain. Also pretty sure it'd cause re-IP'ing of the instances.

But the VPN can come down, and we should definitely clean things up. Even more so, in light of bug 1309874.

Comment 2

[Ed Morley [:emorley]]

(In reply to Kendall Libby [:fubar] from comment #1)
> Despite the DBs being public (a pox on heroku)

Or more a pox on Mozilla not paying for Heroku enterprise/private spaces :-)

Comment 3

[Ed Morley [:emorley]]

But yeah happy to do whatever wrt VPC/subnet/SGs etc.

Comment 4

[Kendall Libby [:fubar] (he/him)]

removed unneeded resources, cleaned up tags. will remove VPN bits when :dcurado has brought the SCL3 side down.

To github.com:mozilla-platform-ops/devservices-aws.git
   abd9ffe..27f1ffd  master -> master

Comment 5

[Kendall Libby [:fubar] (he/him)]

Attachment: devservices-aws PR #8: remove VPN to SCL3

Comment 6

[Kendall Libby [:fubar] (he/him)]

Ed, the treeherder-heroku RDS instance is off in the default VPC instead of the treeherder VPC. Any objections to moving it? I don't think terraform can without nuking it, so it will probably be manual.

Comment 7

[Jake Watkins [:dividehex]]

Comment on attachment 8805641 [details]
devservices-aws PR #8: remove VPN to SCL3

lgtm

Comment 8

[Ed Morley [:emorley]]

(In reply to Kendall Libby [:fubar] from comment #6)
> Ed, the treeherder-heroku RDS instance is off in the default VPC instead of
> the treeherder VPC. Any objections to moving it? I don't think terraform can
> without nuking it, so it will probably be manual.

Go for it :-)

Comment 9

[Kendall Libby [:fubar] (he/him)]

Looks like it can't be moved manually either; the only option is to change security groups. Tried making a read-replica that we could promote and migrate to, but that also fails as you can't specify the db subnet group when creating the replica. Argh.

Comment 10

[Ed Morley [:emorley]]

I'm happy for you to just nuke the treeherder-heroku RDS instance and re-create from the latest snapshot of prod, if it's easier :-)

(Given we probably want to change the name, I guess we could just spin up a new instance, I switch the environment variable for the prototype instance, then we kill the old RDS instance?)

Comment 11

[Kendall Libby [:fubar] (he/him)]

I've sorted out the issue I was having yesterday and have spun up a new dev instance from the latest prod snapshot (rds:treeherder-prod-2016-11-10-07-05). Should be able to just replace treeherder-heroku with treeherder-dev everywhere. 

Let me know when prototype's changed over and it's safe to remove the treeherder-heroku instance.

Comment 12

[Ed Morley [:emorley]]

Is the new treeherder-dev instance using the credentials of prod, or have they been updated after the re-snapshot? (https://github.com/hashicorp/terraform/issues/8604 seems like it would have to be done manually for now)

Comment 13

[Kendall Libby [:fubar] (he/him)]

Still prod credentials; I'm not sure that I have the dev creds.

Comment 14

[Ed Morley [:emorley]]

I have:
* paused celerybeat on treeherder-prototype
* generated new dev credentials
* logged into the treeherder-dev as th_admin using the prod password
* changed the th_admin password using `SET PASSWORD = PASSWORD('...');`
* updated DATABASE_URL on the treeherder-prototype Heroku app to use the new instance's domain name, and new password

This triggered a deploy, incl migration run, and a recent master branch migration (that isn't yet present on prod) is now running.

Once that is complete I'll unpause celerybeat and hand back over to James for him to deploy his feature branch (it will need to be rebased on master before then too).

Comment 15

[Ed Morley [:emorley]]

The migrations failed, have left a comment on bug 1311185 comment 24.

Comment 16

[Ed Morley [:emorley]]

Given bug 1311185 comment 25, we'll need to recreate it once the prod DB has had the last few manual changes made (next few days).

prototype is semi-usable at the moment, since I set IGNORE_PREDEPLOY_ERRORS=1 so it at least deployed and pointed at the new DB, and then re-enabled celerybeat.

In the meantime the old treeherder-heroku RDS instance is now unused and can be destroyed (ingestion was failing on it for the last few days due to it being even further behind with schema changes, so we're no worse off).

Comment 17

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:emorley] from comment #16)
>
> In the meantime the old treeherder-heroku RDS instance is now unused and can
> be destroyed 

Thanks, nuking!  https://github.com/mozilla-platform-ops/devservices-aws/pull/28