1597740 - Upgrade treeherder-prod to db.m5.4xlarge

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[Armen [:armenzg]]

Hi Jake,
Could you please do this first thing this morning?
We can hold off on increasing the IOPS until we get this done sooner.

We've been having tree closures for the last 12 hours.

Thank you!

Comment 1

[Armen [:armenzg]]

Jake: Please coordinate with Kyle, Bob and Cameron. I don't think I'm thinking straight anymore

Comment 2

[Armen [:armenzg]]

dividex: We would like to try this change until we fix the root issue (Current theory is bug 1598091).
Changing to PIOPS helped but we're still under water overnight.

Comment 3

[Jake Watkins [:dividehex]]

Attachment: GitHub Pull Request

Comment 4

[Jake Watkins [:dividehex]]

Upgrade started:
Wed, 20 Nov 2019 20:39:45 GMT Applying modification to database instance class

Comment 5

[Jake Watkins [:dividehex]]

Upgrade is complete

Wed, 20 Nov 2019 20:39:45 GMT	Applying modification to database instance class
Wed, 20 Nov 2019 21:25:05 GMT	DB instance shutdown
Wed, 20 Nov 2019 21:26:28 GMT	Multi-AZ instance failover started.
Wed, 20 Nov 2019 21:26:37 GMT	DB instance restarted
Wed, 20 Nov 2019 21:26:58 GMT	Multi-AZ instance failover completed
Wed, 20 Nov 2019 22:38:39 GMT	Finished applying modification to DB instance class