1469837 - Reset treeherder-stage and treeherder-dev RDS to latest production snapshot

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P1)

Description

[Ed Morley [:emorley]]

Hi Kendall/Jake :-)

Please can the `treeherder-stage` and `treeherder-dev` RDS instances be reset to the latest production snapshot, to roll back a schema change made to them.

Going from bug 1442369 comment 2, I believe the process on your side will be something like:

$ terraform taint aws_db_instance.treeherder-dev-rds
$ terraform taint aws_db_instance.treeherder-stage-rds
$ terraform plan
$ terraform apply

...then after the ~4 mins for destroy and ~15 mins for creation from snapshot, the new instance should be ready (we may also need to check whether it needs that extra reset for the parameter group to take effect).

After that, we'll log in using the admin account password from prod, and change it to match the respective environments DATABASE_URL password, then initiate a new stage/prototype Heroku deploy from `master` (where the schema change will have been reverted). (And also clear the Redis cache for both)

Comment 1

[Ed Morley [:emorley]]

s/extra reset/extra restart/

Comment 2

[Jake Watkins [:dividehex]]

The destruction and recreation of the `treeherder-stage` and `treeherder-dev` has completed.  The parameter groups were pending-reboot so I rebooted them.  Parameter groups are now in-sync.  :-)

Comment 3

[Ed Morley [:emorley]]

Many thanks for the quick response!

I've:
* updated the password on both by connecting using the prod master password and then updating using `SET PASSWORD = '<prior dev/stage password>';`
* initiated a new `master` deploy via Heroku's UI for stage/prototype & re-enabled auto-deploy
* cleared the Redis cache using https://devcenter.heroku.com/articles/heroku-redis#redis-cli and then the Redis command `flushall` on both Redis instances

All seems to be working fine :-)