1413542 - Update Treeherder RDS instances from MySQL 5.7.17 to 5.7.23

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P2)

Description

[Ed Morley [:emorley]]

Not urgent; we should do it some time after the change freeze.

Release notes:
https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-18.html
https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-19.html

Comment 1

[Ed Morley [:emorley]]

Attachment: devservices-aws PR #79: Upgrade treeherder-dev MySQL

Comment 2

[Jake Watkins [:dividehex]]

Comment on attachment 8952087 [details] [review]
devservices-aws PR #79: Upgrade treeherder-dev MySQL

I'm going to 302 to :fubar since I'm swamped with datacenter move deadlines

Comment 3

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8952087 [details] [review]
devservices-aws PR #79: Upgrade treeherder-dev MySQL

Currently applying update; will update when it's finished.

Comment 4

[Kendall Libby [:fubar] (he/him)]

Done.

Comment 5

[Ed Morley [:emorley]]

Many thanks - will leave this bug open to track verifying no issues with 5.7.19 and then updating stage/prod to use it too.

Comment 6

[Ed Morley [:emorley]]

Upgrade log looks fine:
https://emorley.pastebin.mozilla.org/9079148

Comment 7

[Ed Morley [:emorley]]

5.7.21 is now available, so lets target that instead:
https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-20.html
https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-21.html

Comment 8

[Ed Morley [:emorley]]

Attachment: devservices-aws PR #83: Upgrade treeherder-dev MySQL again

Hi! This can be applied at any time - no need for coordination.
Later I'll open another PR for stage/prod (so the bug should stay open).
Many thanks :-)

Comment 9

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8967358 [details] [diff] [review]
devservices-aws PR #83: Upgrade treeherder-dev MySQL again

Review of attachment 8967358 [details] [diff] [review]:
-----------------------------------------------------------------

approved, merged, about to deploy...

Comment 10

[Kendall Libby [:fubar] (he/him)]

aws_db_instance.treeherder-dev-rds: Modifying... (ID: treeherder-dev)
  engine_version: "5.7.19" => "5.7.21"
aws_db_instance.treeherder-dev-rds: Still modifying... (ID: treeherder-dev, 10s elapsed)
aws_db_instance.treeherder-dev-rds: Still modifying... (ID: treeherder-dev, 20s elapsed)
aws_db_instance.treeherder-dev-rds: Still modifying... (ID: treeherder-dev, 30s elapsed)
aws_db_instance.treeherder-dev-rds: Modifications complete after 32s (ID: treeherder-dev)

and I always forget apply_immediately, so manually forced the upgrade via console. and done an hour later, due to making two db backups/snapshots.

Comment 11

[Ed Morley [:emorley]]

Oops meant to come back to this sooner. Now 5.7.23 is available:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.VersionMgmt

https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-22.html
https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-23.html

I'll open a PR to update dev to 5.7.23 and follow up with the others soon after.

Comment 12

[Ed Morley [:emorley]]

Attachment: devservices-aws PR #89: Upgrade treeherder-dev MySQL to 5.7.23

Comment 13

[Ed Morley [:emorley]]

When you apply, could you use apply_immediately? :-)

Comment 14

[Ed Morley [:emorley]]

Attachment: devservices-aws PR #95: Upgrade treeherder-{stage,prod-ro} MySQL to 5.7.23

Please could you apply this when you get a chance (using apply immediately again).
I believe stage should take less time than the previous one, since it's multi-AZ so the upgrade fails over instead of taking it offline.

Comment 15

[Jake Watkins [:dividehex]]

(In reply to Ed Morley [:emorley] from comment #14)

Created attachment 9035280 [details] [review]
devservices-aws PR #95: Upgrade treeherder-{stage,prod-ro} MySQL to 5.7.23

Please could you apply this when you get a chance (using apply immediately again).
I believe stage should take less time than the previous one, since it's multi-AZ so the upgrade fails over instead of taking it offline.

Merged, applied and upgrades completed.
aws_db_instance.treeherder-prod-ro-rds: Modifications complete after 16m18s (ID: treeherder-prod-ro)
aws_db_instance.treeherder-stage-rds: Modifications complete after 56m49s (ID: treeherder-stage)

Comment 16

[Ed Morley [:emorley]]

Thank you :-)

Comment 17

[Ed Morley [:emorley]]

Attachment: devservices-aws PR #96: Upgrade treeherder-prod MySQL to 5.7.23

Hi Jake!

This applies the same change to prod as was applied to dev/stage/prod-ro.

When it was applied for stage it caused 2-5 mins of errors as reported by New Relic which are for tasks that retry, so it shouldn't cause significant impact. However it may be best to apply this at at lower load time of day (eg early pacific) to reduce that further.

Many thanks :-)

Comment 18

[Jake Watkins [:dividehex]]

Comment on attachment 9040971 [details] [review]
devservices-aws PR #96: Upgrade treeherder-prod MySQL to 5.7.23

I'll apply this on Monday 2019/02/11 @ 0900 PST

Comment 19

[Ed Morley [:emorley]]

That sounds great - thank you :-)

Comment 20

[Jake Watkins [:dividehex]]

Comment on attachment 9040971 [details] [review]
devservices-aws PR #96: Upgrade treeherder-prod MySQL to 5.7.23

aws_db_instance.treeherder-prod-rds: Modifications complete after 1h3m21s (ID: treeherder-prod)

Comment 21

[Ed Morley [:emorley]]

Many thanks!

Terraform change landed in:
https://github.com/mozilla-platform-ops/devservices-aws/commit/6db0b6441529dba4a38c023813a6f003b5cf779f

Looking at the AWS RDS event logs, the DB started its restart at 17:38:17 and was back at 17:41:43, so much quicker than stage.