Upgrade addons prod cluster in phx to MySQL 5.5

RESOLVED FIXED

Status

Data & BI Services Team
DB: MySQL
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: sheeri, Unassigned)

Tracking

Details

(Whiteboard: [2013q1])

(Reporter)

Description

6 years ago
Upgrade addons prod cluster in phx to MySQL 5.5
(Reporter)

Updated

6 years ago
Whiteboard: [2013q1]
(Reporter)

Comment 1

6 years ago
Took addons1.db.phx1.mozilla.com out of the load balancer. Connections are draining as we speak. Nothing is slaving addons1 so this is an appropriate first hop to upgrade addons in production.
(Reporter)

Comment 2

6 years ago
exporting all the databases on addons1 now.
(Reporter)

Comment 3

5 years ago
[root@addons1.db.phx1 ~]# time mysqldump --all-databases --routines --triggers --events > `date +%Y-%m-%d`_backup.sql

real    26m31.633s
user    10m49.246s
sys     1m10.818s

[root@addons1.db.phx1 ~]# ls -rlth 2013-03-04_backup.sql 
-rw-r--r-- 1 root root 36G Mar  4 16:59 2013-03-04_backup.sql

localhost:nodes scabral$ svn commit -m "updating addons1 to be mariadb55"
Sending        databases.pp
Transmitting file data .
Committed revision 60081.
(Reporter)

Comment 4

5 years ago
[root@addons1.db.phx1 ~]# rpm -qa | grep -i maria
MariaDB-client-5.5.29-1.x86_64
MariaDB-compat-5.5.29-1.x86_64
MariaDB-server-5.5.29-1.x86_64
MariaDB-common-5.5.29-1.x86_64

[root@addons1.db.phx1 mysql]# time mysql < /root/2013-03-04_backup.sql

Here we go!
(Reporter)

Comment 5

5 years ago
addons1.db.phx1.mozilla.com is updated to MySQL 5.5. Checksum differences are only for the FLOAT/DOUBLE and can be ignored, so I will put addons1 back in the load balancer pool now. Fun times!
(Reporter)

Comment 6

5 years ago
So far, so good. Taking addons2 out of the load balancer for upgrade.
(Reporter)

Comment 7

5 years ago
addons2 has been upgraded to MariaDB 5.5 and the backup is streaming from addons1 now.
(Reporter)

Comment 8

5 years ago
addons2 has been upgraded, and put back into the load balancer.
(Reporter)

Comment 9

5 years ago
Taking adons3 out of the load balancer for upgrade and upgrading.
(Reporter)

Comment 10

5 years ago
addons3 and addons4 have been upgraded. Will work on addons5 and addons6 now.
(Reporter)

Comment 11

5 years ago
addons5 and addons6 have been upgraded. 

All that's left to do is failover and do addons7 (and fail back).
(Reporter)

Comment 12

5 years ago
I have changed all the machines to slave off addons1, so that the move from addons7 to addons1 is a simple failover.
(Reporter)

Comment 13

5 years ago
The failover was complete on Thursday, and addons1 remains the master. I'm doing a final checksum now to ensure everything looks good on addons7 and if it does, I'll switch back.
(Reporter)

Comment 14

5 years ago
checksum came out clean, so I will switch back soon.
(Reporter)

Comment 15

5 years ago
All machines are now slaving off addons7. Nothing left to do but actually do the switch back now!
(Reporter)

Comment 16

5 years ago
And, we've failed back to addons7 as the master. Resolving!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.