783568 - Upgrade mysql to Percona on servers that should be Percona servers

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Eric Ziegenhorn :ericz]

node[36-39].seamicro.phx1 all run MySQL and at least some of those should be Percona.  Please manually upgrade these as it is breaking yum and puppet because Percona-XtraDB-Cluster-shared conflicts with Percona-Server-shared-51.

Comment 1

[Sheeri Cabral [:sheeri]]

Starting with node39:

[[e-mail]1 ~]# rpm -qa | grep -i percona
[[e-mail]1 ~]# rpm -qa | grep -i mysql
mysql-server-5.1.52-1.el6_0.1.i686
mysql-5.1.52-1.el6_0.1.i686
mysql-libs-5.1.52-1.el6_0.1.i686
perl-DBD-MySQL-4.013-3.el6.i686

[[e-mail]1 ~]# rpm -e mysql-server mysql mysql-libs perl-DBD-MySQL
error: Failed dependencies:
        libmysqlclient.so.16 is needed by (installed) postfix-2:2.6.6-2.2.el6_1.i686
        libmysqlclient.so.16(libmysqlclient_16) is needed by (installed) postfix-2:2.6.6-2.2.el6_1.i686
        mysql-libs is needed by (installed) postfix-2:2.6.6-2.2.el6_1.i686

We've forced this before with the postfix dependency and it's been OK:

[[e-mail]1 ~]# rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
warning: /etc/my.cnf saved as /etc/my.cnf.rpmsave

let's preserve that file:

[[e-mail]1 ~]# cp /etc/my.cnf.rpmsave  /etc/my.cnf.good

Comment 2

[Sheeri Cabral [:sheeri]]

[[e-mail]1 ~]# yum install Percona-Server-server-51  Percona-Server-shared-51

.....

Dependencies Resolved

=========================================================================================================================================================================
 Package                                          Arch                         Version                                             Repository                       Size
=========================================================================================================================================================================
Installing:
 Percona-Server-server-51                         i686                         5.1.63-rel13.4.443.rhel6                            percona                          12 M
 Percona-Server-shared-51                         i686                         5.1.63-rel13.4.443.rhel6                            percona                         2.7 M

Transaction Summary
=========================================================================================================================================================================
Install       2 Package(s)

Total download size: 15 M
Installed size: 43 M
Is this ok [y/N]: y

....

Warning: RPMDB altered outside of yum.
** Found 3 pre-existing rpmdb problem(s), 'yum check' output follows:
2:postfix-2.6.6-2.2.el6_1.i686 has missing requires of libmysqlclient.so.16
2:postfix-2.6.6-2.2.el6_1.i686 has missing requires of libmysqlclient.so.16(libmysqlclient_16)
2:postfix-2.6.6-2.2.el6_1.i686 has missing requires of mysql-libs
  Installing : Percona-Server-shared-51-5.1.63-rel13.4.443.rhel6.i686                                                                                                1/2 
  Installing : Percona-Server-server-51-5.1.63-rel13.4.443.rhel6.i686      

  Verifying  : Percona-Server-shared-51-5.1.63-rel13.4.443.rhel6.i686                                                                                                1/2 
  Verifying  : Percona-Server-server-51-5.1.63-rel13.4.443.rhel6.i686                                                                                                2/2 

Installed:
  Percona-Server-server-51.i686 0:5.1.63-rel13.4.443.rhel6                            Percona-Server-shared-51.i686 0:5.1.63-rel13.4.443.rhel6                           

Complete!

But the my.cnf is not in place, so:
[[e-mail]1 ~]# /etc/init.d/mysql stop

...

[[e-mail]1 ~]# cp /etc/my.cnf.good /etc/my.cnf

(also, comment out "basedir" in /etc/my.cnf)

MySQL started OK, but I got "mysql: Command not found" so I am installing the client:

 yum install Percona-server-client-51

Comment 3

[Sheeri Cabral [:sheeri]]

mysql> select @@version, @@hostname;
+--------------------+----------------------------------+
| @@version          | @@hostname                       |
+--------------------+----------------------------------+
| 5.1.63-rel13.4-log | node39.seamicro.phx1.mozilla.com |
+--------------------+----------------------------------+
1 row in set (0.00 sec)

node39 is done. For completion's sake, I'm manually running:

[[e-mail]1 ~]# yum install percona-xtrabackup percona-toolkit

Comment 4

[Sheeri Cabral [:sheeri]]

I'll wait until I get the all-clear that node39 is OK, then I'll do the other nodes.

Comment 5

[Sheeri Cabral [:sheeri]]

Attachment: node39 puppetdashboard shows is OK

Looks like node39 is OK according to this:

https://puppetdashboard.mozilla.org/nodes/407

Comment 6

[Sheeri Cabral [:sheeri]]

Upgrade to Percona 5.1 checklist:

0) take out of load balancer, downtime in nagios, etc
1) /etc/init.d/mysql stop
2) rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
3) cp /etc/my.cnf.rpmsave  /etc/my.cnf.good
4) cp /etc/my.cnf.rpmsave  /etc/my.cnf
5) Make sure the server is config'd in puppet as getting the Percona package
6) yum install Percona-Server-server-51  Percona-Server-shared-51
7) yum install Percona-server-client-51
8) yum install percona-xtrabackup percona-toolkit

Comment 7

[Sheeri Cabral [:sheeri]]

Doing node38 now:

DONE 0) take out of load balancer, downtime in nagios, etc
DONE 1) /etc/init.d/mysql stop
DONE 2) rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
DONE 3) cp /etc/my.cnf.rpmsave  /etc/my.cnf.good
DONE 4) cp /etc/my.cnf.rpmsave  /etc/my.cnf
DONE 5) Make sure the server is config'd in puppet as getting the Percona package
IN PROGRESS 6) yum install Percona-Server-server-51  Percona-Server-shared-51
7) yum install Percona-server-client-51
8) yum install percona-xtrabackup percona-toolkit

Comment 8

[Sheeri Cabral [:sheeri]]

Doing node38 now:

DONE 0) take out of load balancer, downtime in nagios, etc
DONE 1) /etc/init.d/mysql stop
DONE 2) rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
DONE 3) cp /etc/my.cnf.rpmsave  /etc/my.cnf.good
DONE 4) cp /etc/my.cnf.rpmsave  /etc/my.cnf
DONE 5) Make sure the server is config'd in puppet as getting the Percona package
DONE 6) yum install Percona-Server-server-51  Percona-Server-shared-51
DONE 7) yum install Percona-Server-client-51
DONE 8) yum install percona-xtrabackup percona-toolkit
IN PROGRESS 9) run /usr/local/sbin/puppetctl run to make sure there are no errors.

Comment 9

[Sheeri Cabral [:sheeri]]

node38 upgraded!

node39:

Doing node38 now:

DONE 0) take out of load balancer, downtime in nagios, etc
DONE 1) /etc/init.d/mysql stop
DONE 2) rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
DONE 3) cp /etc/my.cnf.rpmsave  /etc/my.cnf.good
DONE 4) cp /etc/my.cnf.rpmsave  /etc/my.cnf
DONE 5) Make sure the server is config'd in puppet as getting the Percona package
IN PROGRESS 6) yum install Percona-Server-server-51  Percona-Server-shared-51
 7) yum install Percona-Server-client-51
 8) yum install percona-xtrabackup percona-toolkit
 9) run /usr/local/sbin/puppetctl run to make sure there are no errors.

Comment 10

[Sheeri Cabral [:sheeri]]

Doing node39 now:

DONE 0) take out of load balancer, downtime in nagios, etc
DONE 1) /etc/init.d/mysql stop
DONE 2) rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
DONE 3) cp /etc/my.cnf.rpmsave  /etc/my.cnf.good
DONE 4) cp /etc/my.cnf.rpmsave  /etc/my.cnf
DONE 5) Make sure the server is config'd in puppet as getting the Percona package
DONE 6) yum install Percona-Server-server-51  Percona-Server-shared-51
DONE 7) yum install Percona-Server-client-51
DONE 8) yum install percona-xtrabackup percona-toolkit
IN PROGRESS 9) run /usr/local/sbin/puppetctl run to make sure there are no errors.

Comment 11

[Sheeri Cabral [:sheeri]]

er, that was all for node37! node39 was done the other day.

Now onto node36, which is the tricky one, as it's the master.

Comment 12

[Sheeri Cabral [:sheeri]]

Talking with mkelly in #affiliates, it's OK to do now:

Doing node39 now:

DONE 0) take out of load balancer, downtime in nagios, etc
DONE 1) /etc/init.d/mysql stop
DONE 2) rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
DONE 3) cp /etc/my.cnf.rpmsave  /etc/my.cnf.good
DONE 4) cp /etc/my.cnf.rpmsave  /etc/my.cnf
DONE 5) Make sure the server is config'd in puppet as getting the Percona package
IN PROGRESS 6) yum install Percona-Server-server-51  Percona-Server-shared-51
 7) yum install Percona-Server-client-51
 8) yum install percona-xtrabackup percona-toolkit
 9) run /usr/local/sbin/puppetctl run to make sure there are no errors.

Comment 13

[Sheeri Cabral [:sheeri]]

node36 progress:

DONE 0) take out of load balancer, downtime in nagios, etc
DONE 1) /etc/init.d/mysql stop
DONE 2) rpm -e --nodeps mysql-server mysql mysql-libs perl-DBD-MySQL
DONE 3) cp /etc/my.cnf.rpmsave  /etc/my.cnf.good
DONE 4) cp /etc/my.cnf.rpmsave  /etc/my.cnf
DONE 5) Make sure the server is config'd in puppet as getting the Percona package
DONE 6) yum install Percona-Server-server-51  Percona-Server-shared-51
DONE 7) yum install Percona-Server-client-51
DONE 8) yum install percona-xtrabackup percona-toolkit
IN PROGRESS 9) run /usr/local/sbin/puppetctl run to make sure there are no errors.

Comment 14

[Sheeri Cabral [:sheeri]]

resolving!