1442369 - Create a new RDS instance from latest prod snapshot to test bug 1367362

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P1)

Description

[Ed Morley [:emorley]]

For bug 1367362 we need bug classification data on prototype, that only exists in production.

Jake/Kendall - please can the `treeherder-dev` RDS instance (used by the Heroku app treeherder-prototype) be reset to the most recent `treeherder-prod` RDS instance snapshot?

Since we don't yet have bug 1321294 a snapshot will need to be selected manually.

No timing coordination required; I can fix up the credentials etc afterwards (short term breakage of the dev instance is fine).

Many thanks!

Comment 1

[Ed Morley [:emorley]]

I ended up creating a new temporary RDS instance to unblock bug 1367362's demo today (our Outreachy candidate's last day is today). I've added a PR to bug 1321294 with a longer term solution for resetting - at some point after that lands I'll open a new bug to actually do the reset of treeherder-dev.


I'll leave this bug open to track the cleanup of the new temporary RDS instance once bug 1367362 is merged (should a few days to a week).

Comment 2

[Kendall Libby [:fubar] (he/him)]

I had missed the previous update before running with the bug. Ed says that's ok, though.

Process is pretty easy now:

$ terraform taint aws_db_instance.treeherder-dev-rds
$ terraform plan
$ terraform apply

The rds instance is deleted here. since this is where I suddenly saw #c1, I control-C'ed the process. Subsequently, I had to re-run the plan/apply to create the new instance. I'm not sure if that would normally be necessary or not, but running it twice is still easier than generating a PR.

For timing, it takes ~4 minutes to destroy the RDS instance, and ~15 minutes to create from a snapshot.

Also worth noting that the new instance comes up with the password from the snapshot and should be changed manually.

Comment 3

[Kendall Libby [:fubar] (he/him)]

Failed to add - treeherder-dev is reset.

Comment 4

[Ed Morley [:emorley]]

Many thanks for doing that - I've continued the conversation about the reset in bug 1321294.

Re this (morphed) bug - I've now pointed the prototype Heroku app back at `treeherder-dev` RDS, and have deleted the `treeherder-dev-bug1367362` instance.