Closed Bug 828043 Opened 11 years ago Closed 11 years ago

Upgrade addons prod cluster in phx to MySQL 5.5

Categories

(Data & BI Services Team :: DB: MySQL, task)

Product:
Data & BI Services Team
Component:
DB: MySQL
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: scabral, Unassigned)

References

Details

(Whiteboard: [2013q1]) 

Upgrade addons prod cluster in phx to MySQL 5.5
Whiteboard: [2013q1]
Took addons1.db.phx1.mozilla.com out of the load balancer. Connections are draining as we speak. Nothing is slaving addons1 so this is an appropriate first hop to upgrade addons in production.
exporting all the databases on addons1 now.
[root@addons1.db.phx1 ~]# time mysqldump --all-databases --routines --triggers --events > `date +%Y-%m-%d`_backup.sql

real    26m31.633s
user    10m49.246s
sys     1m10.818s

[root@addons1.db.phx1 ~]# ls -rlth 2013-03-04_backup.sql 
-rw-r--r-- 1 root root 36G Mar  4 16:59 2013-03-04_backup.sql

localhost:nodes scabral$ svn commit -m "updating addons1 to be mariadb55"
Sending        databases.pp
Transmitting file data .
Committed revision 60081.
[root@addons1.db.phx1 ~]# rpm -qa | grep -i maria
MariaDB-client-5.5.29-1.x86_64
MariaDB-compat-5.5.29-1.x86_64
MariaDB-server-5.5.29-1.x86_64
MariaDB-common-5.5.29-1.x86_64

[root@addons1.db.phx1 mysql]# time mysql < /root/2013-03-04_backup.sql

Here we go!
addons1.db.phx1.mozilla.com is updated to MySQL 5.5. Checksum differences are only for the FLOAT/DOUBLE and can be ignored, so I will put addons1 back in the load balancer pool now. Fun times!
So far, so good. Taking addons2 out of the load balancer for upgrade.
addons2 has been upgraded to MariaDB 5.5 and the backup is streaming from addons1 now.
addons2 has been upgraded, and put back into the load balancer.
Taking adons3 out of the load balancer for upgrade and upgrading.
addons3 and addons4 have been upgraded. Will work on addons5 and addons6 now.
addons5 and addons6 have been upgraded. 

All that's left to do is failover and do addons7 (and fail back).
I have changed all the machines to slave off addons1, so that the move from addons7 to addons1 is a simple failover.
The failover was complete on Thursday, and addons1 remains the master. I'm doing a final checksum now to ensure everything looks good on addons7 and if it does, I'll switch back.
checksum came out clean, so I will switch back soon.
All machines are now slaving off addons7. Nothing left to do but actually do the switch back now!
And, we've failed back to addons7 as the master. Resolving!
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.