650215 - Enable two-way master-master replication for the bugs01 DB cluster between SJC and PHX

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dave Miller [:justdave]]

In order to have easy fallback to SJC in case anything goes wrong in PHX we need to be replicating in both directions.

1) "grant replication slave on *.* to repl@10.2.70.138 ...."
2) Stop replication on tp-bugs01-master01
3) Ensure replication is caught up on tp-bugs01-slave01
4) "show slave status" on tp-bugs01-slave01
5) "change master to" on tm-bugs01-master01 using the details from
   tp-bugs01-slave01
6) "start slave" on both tm- and tp- bugs01-master01

That's it.

Safeguards:
1) *ALL* traffic to both DB servers from accounts with write access is allowed only via the Zeus load balancers (already done)
2) The R/W vIP on both load balancers will always point at the same master, and only one of them.  This ensures that only one master can be written to at any given time (this way we don't need to worry about avoiding autoincrement conflicts and so forth)

Once this is set up, switching from one master to the other is done by:
1) drain the current master on both Zeus balancers
2) ensure all connections from non-replication users are gone (probably require a minute or so of downtime page put up on the website)
3) enable the other master on both balancers.

Comment 1

[Dave Miller [:justdave]]

This is done.  Surprisingly easy and uneventful. :)