Upgrade addons prod cluster in phx to MySQL 5.5
Took addons1.db.phx1.mozilla.com out of the load balancer. Connections are draining as we speak. Nothing is slaving addons1 so this is an appropriate first hop to upgrade addons in production.
exporting all the databases on addons1 now.
[email@example.com ~]# time mysqldump --all-databases --routines --triggers --events > `date +%Y-%m-%d`_backup.sql real 26m31.633s user 10m49.246s sys 1m10.818s [firstname.lastname@example.org ~]# ls -rlth 2013-03-04_backup.sql -rw-r--r-- 1 root root 36G Mar 4 16:59 2013-03-04_backup.sql localhost:nodes scabral$ svn commit -m "updating addons1 to be mariadb55" Sending databases.pp Transmitting file data . Committed revision 60081.
[email@example.com ~]# rpm -qa | grep -i maria MariaDB-client-5.5.29-1.x86_64 MariaDB-compat-5.5.29-1.x86_64 MariaDB-server-5.5.29-1.x86_64 MariaDB-common-5.5.29-1.x86_64 [firstname.lastname@example.org mysql]# time mysql < /root/2013-03-04_backup.sql Here we go!
addons1.db.phx1.mozilla.com is updated to MySQL 5.5. Checksum differences are only for the FLOAT/DOUBLE and can be ignored, so I will put addons1 back in the load balancer pool now. Fun times!
So far, so good. Taking addons2 out of the load balancer for upgrade.
addons2 has been upgraded to MariaDB 5.5 and the backup is streaming from addons1 now.
addons2 has been upgraded, and put back into the load balancer.
Taking adons3 out of the load balancer for upgrade and upgrading.
addons3 and addons4 have been upgraded. Will work on addons5 and addons6 now.
addons5 and addons6 have been upgraded. All that's left to do is failover and do addons7 (and fail back).
I have changed all the machines to slave off addons1, so that the move from addons7 to addons1 is a simple failover.
The failover was complete on Thursday, and addons1 remains the master. I'm doing a final checksum now to ensure everything looks good on addons7 and if it does, I'll switch back.
checksum came out clean, so I will switch back soon.
All machines are now slaving off addons7. Nothing left to do but actually do the switch back now!
And, we've failed back to addons7 as the master. Resolving!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.