Last Comment Bug 560659 - [tracking] AMO DB uplift
: [tracking] AMO DB uplift
Status: RESOLVED FIXED
[1/11 @ 4PM]
:
Product: mozilla.org Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All Other
: -- normal (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
:
Mentors:
: 581300 (view as bug list)
Depends on: 572518
Blocks: 530005 599922 610299
  Show dependency treegraph
 
Reported: 2010-04-20 14:07 PDT by matthew zeier [:mrz]
Modified: 2015-03-12 08:17 PDT (History)
6 users (show)
phong: needs‑downtime+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description matthew zeier [:mrz] 2010-04-20 14:07:26 PDT

    
Comment 1 timellis 2010-04-21 10:18:51 PDT
There are some servers in MPT for AMO. Create another AMO cluster, during outage, fail to those servers. Then old AMO hardware is available for re-use.
Comment 2 timellis 2010-06-09 10:12:24 PDT
There are actual prod problems with AMO right now.
Comment 3 timellis 2010-06-14 12:58:04 PDT
How to PXE boot a machine:

(1) ssh mozillaadmin@10.2.8.101 ((bizarro password typed next up))
(2) show server names
(3) connect server X (where X is the slot number) will connect to a blade
(4) power on <--power on, wait ~30 seconds
(5) vsp <--console
(6) Press ENTER for OS install menu
(7) <osNum> text console=ttyS0 hostname=<hostname>
Comment 4 timellis 2010-06-14 17:03:00 PDT
These are the boxes slated for AMO02:

10.09  	tm-sfx01-slave02  	2634  	HP - BL460c G6 (1x L5520 24GB BBWC) 	None  	10.2.8.101
10.10 	tm-amo01-slave04 	2635 	HP - BL460c G6 (1x L5520 24GB BBWC) 	infra : database 	10.2.8.101
10.11 	tm-amo02-slave02 	2636 	HP - BL460c G6 (1x L5520 24GB BBWC) 	None 	10.2.8.101
10.12 	tm-amo02-master01 	2637 	HP - BL460c G6 (1x L5520 24GB BBWC) 	None 	10.2.8.101
10.13 	tm-amo02-slave01 	2638 	HP - BL460c G6 (1x L5520 24GB BBWC) 	None 	10.2.8.101

I think to use tm-amo01-slave04 to build the other 3 servers (if not from Backups).
Comment 5 timellis 2010-06-15 19:13:20 PDT
Use bbcp to create new slaves. Old way is super-slow. Be sure my.cnf exists on slave and is valid for that slave before deploying slave.
Comment 6 timellis 2010-07-15 10:21:50 PDT
read only mode is enabled by putting this at the very bottom of settings_local.py:
           read_only_mode(globals())
and restarting apache.

Then I assume I pull every slave out of the pool but one, rebuild the rest of the cluster, bring it back into read/write mode, then pull out the R/O slave and rebuild it while AMO is back online.
Comment 7 Wil Clouser [:clouserw] 2010-07-15 10:24:05 PDT
Sounds reasonable.  I don't know if AMO will run on one slave, but y'all are a better judge of that than me.
Comment 8 timellis 2010-08-18 10:13:16 PDT
*** Bug 581300 has been marked as a duplicate of this bug. ***
Comment 9 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-09-16 23:34:46 PDT
This is ready to go.

The new slaves have been built and already added to the existing pool, we can pull the old slaves out whenever we feel like it.  The new master is currently acting as a slave to the existing master.  We'll move it into place via slave promotion.  This will probably require dropping AMO into read-only mode for about 5 minutes to make sure all the replication pointers sync up before making the switch.
Comment 10 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-10-06 20:39:02 PDT
I'd like to get the VIP for this moved into Zeus to match the setup in PHX while we're doing this.  Currently we have VIPs both in the Netscaler and on the ACE, and both are actually being used (not sure by what).  I think the actual addons site might be using one and samo/vamo using the other?  Would be good to get them all pointed to the same place.  Right now it's a pain when I want to take a slave out because there's more than one place it has to be disabled.
Comment 11 matthew zeier [:mrz] 2010-10-14 21:37:27 PDT
Did this happen today?
Comment 12 matthew zeier [:mrz] 2010-11-09 17:00:29 PST
Is this in progress?  Done?
Comment 13 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-11-09 19:52:03 PST
I must have missed the bugmail when this date was set on the whiteboard and for some reason failed to notice it since then.  Today is my birthday and I'm not even supposed to be here today, would have vetoed it way back then if I'd noticed. No, this hasn't been touched.
Comment 14 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-11-09 19:58:15 PST
I don't know that this needs to be coordinated with another downtime, the required time for it is minimal, and we've had network failures that have taken the site out for longer than this will.  The current master just needs to be dropped into read-only mode long enough for the replication pointers to sync up on all of the slaves (which is probably a matter of seconds in most cases), then change the config on the app to talk to the new database VIP for the master.
Comment 15 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-11-18 19:30:02 PST
OK, all of the slaves have now been re-parented.  The app is still talking to the original master, but the only thing slaving from it now is the new master.  I'd prefer the app config changes be made by oremj because it looks like stuff is still using IP addresses instead of the fake hostnames, so I'm not sure how that'll break pheonix, or how many places it'll need to be changed, etc. For example the IP address of the old master appears in at least 8 files on mradm02 that I've found so far.

The new database IPs should be:

read-write: 10.2.70.12
read-only:  10.2.70.19
Comment 16 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-11-18 19:57:52 PST
btw: I'm happy to do this myself if someone gives me a good complete list of all of the places it actually needs to be changed.  I'd appreciate having app folks on hand when it happens though ;)  More eyes to find if anything breaks, etc.
Comment 17 Jeremy Orem [:oremj] 2010-11-19 15:33:53 PST
Let's switch the IPs on Tuesday. I'm scared.
Comment 18 Jeremy Orem [:oremj] 2010-11-23 14:52:54 PST
So it is safe to change to these IPs at any time?
Comment 19 matthew zeier [:mrz] 2010-12-03 12:36:51 PST
Can you set a date to get this wrapped up?
Comment 20 Jeremy Orem [:oremj] 2010-12-08 11:28:34 PST
Need an answer to comment 18.
Comment 21 matthew zeier [:mrz] 2010-12-09 10:26:48 PST
over to dave for comment.
Comment 22 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-12-09 19:05:08 PST
Should be safe to switch them, the IPs are active.  If you can run a test connection from the mysql command line from one of the webheads and it works then you're good.  If anything we may need firewall poked, that might be the only blocker to it.  You'll be able to figure that out testing from a webhead I'd imagine.
Comment 23 Jeremy Orem [:oremj] 2010-12-22 15:00:49 PST
Will plan on doing this next week.
Comment 24 Jeremy Orem [:oremj] 2011-01-11 16:36:18 PST
I've switched the IPs for SAMO and AMO.

Note You need to log in before you can comment on or make changes to this bug.