1331397 - Reset prototype instance to production database

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[James Graham [:jgraham]]

wlach landed a large number of schema changes to stage/prod recently. In order to simulate deployment as closely as possible I'd like to reset prototype to current prod so that I can avoid running schema changes from upstream and test that my migrations work well.

Comment 1

[Ed Morley [:emorley]]

I've set SKIP_PREDEPLOY=1 on the treeherder-prototype Heroku app, to prevent the custom branch's migrations being run against the new DB (since they need to be rebased against master first), and have disabled auto-deploy and manually deployed master.

Once the new RDS instance is ready (and the credentials updated away from what was in the prod snapshot), we can update DATABASE_URL and remove SKIP_PREDEPLOY.

Comment 2

[Kendall Libby [:fubar] (he/him)]

https://github.com/mozilla-platform-ops/devservices-aws/pull/32 sets it to the lastest prod snapshot, and it has been deployed

Comment 3

[Ed Morley [:emorley]]

I'm unable to connect to the reset treeherder-dev instance using any of these passwords: {previous dev, current stage, current prod, former treeherder-heroku}

Was the password changed after the snapshot restore? If so, what to?

Comment 4

[Kendall Libby [:fubar] (he/him)]

I believe the reset did not successfully finish. The deploy was also updating the stage storage which timed out, and while it looked like the dev reset finished there may have been steps that didn't not complete/happen because the other update failed to finish in time. Would have to look through the terraform internals to know for certain.

In any case, terraform plan says it's not done, so another deploy in progress.

Comment 5

[Kendall Libby [:fubar] (he/him)]

Dev recreated from the treeherder-prod-2017-01-24-07-06 snapshot. Ed, can you check the credentials? We ran into https://github.com/hashicorp/terraform/issues/5417 and dev was completely nuked, so I suspect it has prod's creds.

Comment 6

[Ed Morley [:emorley]]

The reset dev instance had prod's admin user credentials, so I reset them back to the former dev ones, using:

SET PASSWORD = PASSWORD('...');

(For anyone else doing this in the future, look them up via the existing heroku DATABASE_URL vars)

I have also:
* deployed latest master to the Heroku treeherder-prototype app
* enabled auto-deploy from master
* removed the `SKIP_PREDEPLOY=1` env var, so the new migrations on master run
* cleared the memcachier cache

However there are both cloudamqp and memcachier errors now, which I'll break out into a new bug.

Comment 7

[Ed Morley [:emorley]]

The issues in bug 1334087 are now resolved. The treeherder-prototype app is ingesting data successfully now.

James, I'll leave it to you to reset the Elasticsearch content as appropriate, such that it matches the new DB content.