Closed
Bug 828043
Opened 12 years ago
Closed 12 years ago
Upgrade addons prod cluster in phx to MySQL 5.5
Categories
(Data & BI Services Team :: DB: MySQL, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: scabral, Unassigned)
References
Details
(Whiteboard: [2013q1])
Upgrade addons prod cluster in phx to MySQL 5.5
Reporter | ||
Updated•12 years ago
|
Whiteboard: [2013q1]
Reporter | ||
Comment 1•12 years ago
|
||
Took addons1.db.phx1.mozilla.com out of the load balancer. Connections are draining as we speak. Nothing is slaving addons1 so this is an appropriate first hop to upgrade addons in production.
Reporter | ||
Comment 2•12 years ago
|
||
exporting all the databases on addons1 now.
Reporter | ||
Comment 3•12 years ago
|
||
[root@addons1.db.phx1 ~]# time mysqldump --all-databases --routines --triggers --events > `date +%Y-%m-%d`_backup.sql
real 26m31.633s
user 10m49.246s
sys 1m10.818s
[root@addons1.db.phx1 ~]# ls -rlth 2013-03-04_backup.sql
-rw-r--r-- 1 root root 36G Mar 4 16:59 2013-03-04_backup.sql
localhost:nodes scabral$ svn commit -m "updating addons1 to be mariadb55"
Sending databases.pp
Transmitting file data .
Committed revision 60081.
Reporter | ||
Comment 4•12 years ago
|
||
[root@addons1.db.phx1 ~]# rpm -qa | grep -i maria
MariaDB-client-5.5.29-1.x86_64
MariaDB-compat-5.5.29-1.x86_64
MariaDB-server-5.5.29-1.x86_64
MariaDB-common-5.5.29-1.x86_64
[root@addons1.db.phx1 mysql]# time mysql < /root/2013-03-04_backup.sql
Here we go!
Reporter | ||
Comment 5•12 years ago
|
||
addons1.db.phx1.mozilla.com is updated to MySQL 5.5. Checksum differences are only for the FLOAT/DOUBLE and can be ignored, so I will put addons1 back in the load balancer pool now. Fun times!
Reporter | ||
Comment 6•12 years ago
|
||
So far, so good. Taking addons2 out of the load balancer for upgrade.
Reporter | ||
Comment 7•12 years ago
|
||
addons2 has been upgraded to MariaDB 5.5 and the backup is streaming from addons1 now.
Reporter | ||
Comment 8•12 years ago
|
||
addons2 has been upgraded, and put back into the load balancer.
Reporter | ||
Comment 9•12 years ago
|
||
Taking adons3 out of the load balancer for upgrade and upgrading.
Reporter | ||
Comment 10•12 years ago
|
||
addons3 and addons4 have been upgraded. Will work on addons5 and addons6 now.
Reporter | ||
Comment 11•12 years ago
|
||
addons5 and addons6 have been upgraded.
All that's left to do is failover and do addons7 (and fail back).
Reporter | ||
Comment 12•12 years ago
|
||
I have changed all the machines to slave off addons1, so that the move from addons7 to addons1 is a simple failover.
Reporter | ||
Comment 13•12 years ago
|
||
The failover was complete on Thursday, and addons1 remains the master. I'm doing a final checksum now to ensure everything looks good on addons7 and if it does, I'll switch back.
Reporter | ||
Comment 14•12 years ago
|
||
checksum came out clean, so I will switch back soon.
Reporter | ||
Comment 15•12 years ago
|
||
All machines are now slaving off addons7. Nothing left to do but actually do the switch back now!
Reporter | ||
Comment 16•12 years ago
|
||
And, we've failed back to addons7 as the master. Resolving!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Updated•10 years ago
|
Product: mozilla.org → Data & BI Services Team
You need to log in
before you can comment on or make changes to this bug.
Description
•