741099 - Set up new bedrock cluster with Percona 5.5 and get replication going between them.

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Sheeri Cabral [:sheeri]]

bedrock1.db.scl3.mozilla.com
bedrock2.db.scl3.mozilla.com

Comment 1

[Sheeri Cabral [:sheeri]]

Put into puppet as a percona55 server.

Comment 2

[Sheeri Cabral [:sheeri]]

Had do to stuff puppet should have done (in  order):
yum install Percona-Server-server-55 Percona-Server-shared-compat Percona-Server-client-55 percona-toolkit perl-DBD-mysql

 mkdir /var/log/mysql; chown -R mysql:mysql /var/log/mysql

put in grants for nagiosdaemon

--------

Manual stuff I did:
start mysql
delete all users except root@localhost
set root password
create a /root/.my.cnf
set replication password
set nagios grants
set replication going

The config file for bedrock1 is completely wrong, but bedrock2 is mostly right, except for not inheriting the proper read_only=ON setting.

Comment 3

[Sheeri Cabral [:sheeri]]

punting to jabba because the config file for bedrock1 is totally wrong (and I can't do replication without, say, binlogs)

Comment 4

[Justin Dow [:jabba]]

There were some errors in server.pp that were causing things to fail. I fixed them. Here is the diff:

Index: server.pp
===================================================================
--- server.pp	(revision 32504)
+++ server.pp	(revision 32508)
@@ -18,8 +18,8 @@
     }
 
     $package_name = $package_type ? {
-        "percona55" => ["Percona-Server-server-55", "Percona-Server-shared-compat", "Peronca-Server-client-55", "percona-toolkit", "perl-DBD-mysql"],
-        "percona51" => ["Percona-Server-server-51", "Percona-Server-shared-compat", "Peronca-Server-client-51", "percona-toolkit", "perl-DBD-mysql"],
+        "percona55" => ["Percona-Server-server-55", "Percona-Server-shared-compat", "Percona-Server-client-55", "percona-toolkit", "perl-DBD-MySQL"],
+        "percona51" => ["Percona-Server-server-51", "Percona-Server-shared-compat", "Percona-Server-client-51", "percona-toolkit", "perl-DBD-MySQL"],
         default => "mysql-server",
     }
     $service_name = $package_type ? {
@@ -36,7 +36,7 @@
     }
 
     package {
-        "${package_name}":
+        $package_name:
             ensure => present,
             require => $package_type ? {
                 "percona55" => Yumrepo["percona"],
@@ -119,7 +119,7 @@
                 unless      => "test -f /root/.my.cnf",
                 path => ["/bin", "/usr/bin", "/usr/local/bin"],
                 command     => "mysqladmin -u root password \"${password}\"",
-                require     => Service["${service_name}"],
+                require     => Service[$service_name],
                 before      => Class["mysql2::client"];
         }
 
@@ -136,21 +136,21 @@
             owner           => "mysql",
             group           => "mysql",
             content         => template("mysql2/my.cnf.erb"),
-            notify          => Service["${service_name}"];
+            notify          => Service[$service_name];
 
         "/var/lib/mysql-innodb":
             ensure => directory,
             owner => "mysql",
             group => "mysql",
-            before => Service["${service_name}"];
+            before => Service[$service_name];
     }
 
     service {
-        "$service_name":
+        $service_name:
             ensure => running,
             enable => true,
             hasrestart => true,
             hasstatus => true,
-            require => Package["${package_name}"];
+            require => Package[$package_name];
     }
 }

Comment 5

[Justin Dow [:jabba]]

Also, hold before putting this into production. bedrock1 shows:

[e-mail]3:~> hpacucli controller slot=0 show config

Smart Array P410i in Slot 0 (Embedded)    (sn: 5001438018EC9860)

   array A (Solid State SATA, Unused Space: 0 MB)


      logicaldrive 1 (223.5 GB, RAID 1, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, Solid State SATA, 240.0 GB, Predictive Failure)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, Solid State SATA, 240.0 GB, Predictive Failure)

   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250 (WWID: 5001438018EC986F)

Comment 6

[Justin Dow [:jabba]]

Please evaluate the current configs and let me know what else I should put into puppet to make this fully automated next time.

Comment 7

[Sheeri Cabral [:sheeri]]

Things look good now, everything's working as planned, now that you've fixed it and ensured that MySQL doesn't restart when a config parameter is changed.