1597699 - Change treeherder-prod storage type to Provisioned IOPS

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, task)

Description

[Armen [:armenzg]]

We've been closing the trees daily, specially, when cycle_data starts doing clean ups.
For now let's increase the storage another 350GB which should give us another ~1,000 IOPS (Storage rate $0.23 per GB-month -> 350 * .23 = $81).

Comment 1

[Kyle Lahnakoski [:ekyle]]

If the instance type can be change so that IOPS increases without increasing the database size, that would be best. Also, maybe we can just pay for more IOPS?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I don't see why we wouldn't pay for it, keeping things running is priority #1, hopefully in Q1 we can find a way to reduce it back down.

Comment 3

[Kendall Libby [:fubar] (he/him)]

https://bugzilla.mozilla.org/show_bug.cgi?id=1566204#c3

Comment 4

[Jake Watkins [:dividehex]]

dividehex 11:30 AM
(my 2 cents)  I think it is pretty clear we should up the IOPs.  The only question is do we up it by uping the gp2 size or do we up it to piops and eat the cost.  I personally think we should move to piops since it actually designed for IO intensive workloads, which is what we have here.
Armen (armenzg) 11:30 AM
@dividehex I agree with you
fubar:spiral_calendar_pad: 11:30 AM
wfm
dividehex 11:30 AM
I'm not the decision maker on that though
^^ that guy is :simple_smile:
fubar:spiral_calendar_pad: 11:31 AM
shipit
dividehex 11:31 AM
makes it so

Comment 5

[Jake Watkins [:dividehex]]

Attachment: GitHub Pull Request

Comment 6

[Jake Watkins [:dividehex]]

Terraform will perform the following actions:

  ~ aws_db_instance.treeherder-prod-rds
      iops:         "0" => "7000"
      storage_type: "gp2" => "io1"

Tue, 19 Nov 2019 19:49:49 GMT	Finished preparing for modification of storage
Tue, 19 Nov 2019 19:53:53 GMT	Applying modification to allocated storage
Tue, 19 Nov 2019 19:56:58 GMT	Finished applying modification to allocated storage

Comment 7

[Jake Watkins [:dividehex]]

Prod is currently in the Storage-optimization state.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html

Comment 8

[Armen [:armenzg]]

Thank you!
Things look a bit better overnight.