1567257 - Database changes to increase IOPS cap

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, task)

Description

[Armen [:armenzg]]

We want to try to increase the IOPS cap of the treeherder-prod database by increasing its database size instead of switching to the Provisioned IOPS storage. The latter is far more expensive (about $1,000/month difference for 1TB).

Increasing the storage size to increase the IOPS cap might just be sufficient.

Let's give it a try!

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-platform-ops/devservices-aws/pull/108

Comment 2

[Jake Watkins [:dividehex]]

Increasing the storage size is a fine alternative in this situation but I want everyone to beware we cannot downsize rds storage cap once it has been scaled up. So this change will be permanent on the prod instance.

Comment 3

[Armen [:armenzg]]

Go for it. Thanks!

Comment 4

[Jake Watkins [:dividehex]]

-dev has completed and is now rolled back to gp2. -prod-ro is still modifying

Mon, 22 Jul 2019 16:15:02 GMT Preparing for modification of storage
Mon, 22 Jul 2019 16:16:05 GMT Finished preparing for modification of storage
Mon, 22 Jul 2019 16:16:20 GMT DB instance shutdown

Comment 5

[Jake Watkins [:dividehex]]

-prod-ro has completed
Mon, 22 Jul 2019 16:16:03 GMT Applying modification to allocated storage
Mon, 22 Jul 2019 20:35:19 GMT Finished applying modification to allocated storage

Comment 6

[Jake Watkins [:dividehex]]

-prod storage cap increase started

Tue, 23 Jul 2019 15:58:30 GMT Applying modification to allocated storage

Comment 7

[Armen [:armenzg]]

Attachment: treeherder-prod - SUM of Read & Write IOPS (after changes!)

This graph shows that we're using above 3,000 IOPS.

Let's hope this was it! Thanks Jake!

Comment 8

[Jake Watkins [:dividehex]]

For the -prod record:
Tue, 23 Jul 2019 15:58:30 GMT Applying modification to allocated storage
Tue, 23 Jul 2019 18:34:45 GMT Finished applying modification to allocated storage