Closed Bug 560659 Opened 14 years ago Closed 13 years ago

[tracking] AMO DB uplift

Categories

(mozilla.org Graveyard :: Server Operations, task)

All
Other
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mrz, Assigned: oremj)

References

Details

(Whiteboard: [1/11 @ 4PM])

      No description provided.
Assignee: server-ops → tellis
There are some servers in MPT for AMO. Create another AMO cluster, during outage, fail to those servers. Then old AMO hardware is available for re-use.
There are actual prod problems with AMO right now.
Severity: minor → critical
How to PXE boot a machine:

(1) ssh mozillaadmin@10.2.8.101 ((bizarro password typed next up))
(2) show server names
(3) connect server X (where X is the slot number) will connect to a blade
(4) power on <--power on, wait ~30 seconds
(5) vsp <--console
(6) Press ENTER for OS install menu
(7) <osNum> text console=ttyS0 hostname=<hostname>
These are the boxes slated for AMO02:

10.09  	tm-sfx01-slave02  	2634  	HP - BL460c G6 (1x L5520 24GB BBWC) 	None  	10.2.8.101
10.10 	tm-amo01-slave04 	2635 	HP - BL460c G6 (1x L5520 24GB BBWC) 	infra : database 	10.2.8.101
10.11 	tm-amo02-slave02 	2636 	HP - BL460c G6 (1x L5520 24GB BBWC) 	None 	10.2.8.101
10.12 	tm-amo02-master01 	2637 	HP - BL460c G6 (1x L5520 24GB BBWC) 	None 	10.2.8.101
10.13 	tm-amo02-slave01 	2638 	HP - BL460c G6 (1x L5520 24GB BBWC) 	None 	10.2.8.101

I think to use tm-amo01-slave04 to build the other 3 servers (if not from Backups).
Use bbcp to create new slaves. Old way is super-slow. Be sure my.cnf exists on slave and is valid for that slave before deploying slave.
Depends on: 572518
Group: infra
read only mode is enabled by putting this at the very bottom of settings_local.py:
           read_only_mode(globals())
and restarting apache.

Then I assume I pull every slave out of the pool but one, rebuild the rest of the cluster, bring it back into read/write mode, then pull out the R/O slave and rebuild it while AMO is back online.
Whiteboard: Needs outage: 24 July 2010, NOON.
Sounds reasonable.  I don't know if AMO will run on one slave, but y'all are a better judge of that than me.
Whiteboard: Needs outage: 24 July 2010, NOON. → [blocked]
Whiteboard: [blocked]
Assignee: tellis → mrz
Assignee: mrz → justdave
This is ready to go.

The new slaves have been built and already added to the existing pool, we can pull the old slaves out whenever we feel like it.  The new master is currently acting as a slave to the existing master.  We'll move it into place via slave promotion.  This will probably require dropping AMO into read-only mode for about 5 minutes to make sure all the replication pointers sync up before making the switch.
Flags: needs-downtime+
Whiteboard: 09/21/2010
Blocks: 599922
Whiteboard: 09/21/2010 → 09/30/2010 @ 4pm
Whiteboard: 09/30/2010 @ 4pm → 10/07/2010 @ 4pm
I'd like to get the VIP for this moved into Zeus to match the setup in PHX while we're doing this.  Currently we have VIPs both in the Netscaler and on the ACE, and both are actually being used (not sure by what).  I think the actual addons site might be using one and samo/vamo using the other?  Would be good to get them all pointed to the same place.  Right now it's a pain when I want to take a slave out because there's more than one place it has to be disabled.
Whiteboard: 10/07/2010 @ 4pm → 10/14/2010 @ 4pm
Did this happen today?
Whiteboard: 10/14/2010 @ 4pm → 11/09/2010 @ 4pm
Blocks: 610299
Is this in progress?  Done?
I must have missed the bugmail when this date was set on the whiteboard and for some reason failed to notice it since then.  Today is my birthday and I'm not even supposed to be here today, would have vetoed it way back then if I'd noticed. No, this hasn't been touched.
I don't know that this needs to be coordinated with another downtime, the required time for it is minimal, and we've had network failures that have taken the site out for longer than this will.  The current master just needs to be dropped into read-only mode long enough for the replication pointers to sync up on all of the slaves (which is probably a matter of seconds in most cases), then change the config on the app to talk to the new database VIP for the master.
Whiteboard: 11/09/2010 @ 4pm → 11/18/2010 @ 6pm
OK, all of the slaves have now been re-parented.  The app is still talking to the original master, but the only thing slaving from it now is the new master.  I'd prefer the app config changes be made by oremj because it looks like stuff is still using IP addresses instead of the fake hostnames, so I'm not sure how that'll break pheonix, or how many places it'll need to be changed, etc. For example the IP address of the old master appears in at least 8 files on mradm02 that I've found so far.

The new database IPs should be:

read-write: 10.2.70.12
read-only:  10.2.70.19
Assignee: justdave → jeremy.orem+bugs
btw: I'm happy to do this myself if someone gives me a good complete list of all of the places it actually needs to be changed.  I'd appreciate having app folks on hand when it happens though ;)  More eyes to find if anything breaks, etc.
Flags: needs-downtime+
Whiteboard: 11/18/2010 @ 6pm
Let's switch the IPs on Tuesday. I'm scared.
So it is safe to change to these IPs at any time?
Can you set a date to get this wrapped up?
Need an answer to comment 18.
over to dave for comment.
Assignee: jeremy.orem+bugs → justdave
Should be safe to switch them, the IPs are active.  If you can run a test connection from the mysql command line from one of the webheads and it works then you're good.  If anything we may need firewall poked, that might be the only blocker to it.  You'll be able to figure that out testing from a webhead I'd imagine.
Assignee: justdave → jeremy.orem+bugs
Will plan on doing this next week.
Severity: critical → normal
Whiteboard: [12/20 @ 4PM]
Flags: needs-downtime+
Whiteboard: [12/20 @ 4PM] → [12/30 @ 4PM]
Whiteboard: [12/30 @ 4PM] → [1/11 @ 4PM]
I've switched the IPs for SAMO and AMO.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.