Closed Bug 721516 Opened 13 years ago Closed 13 years ago

migrate seamonkey systems out of sjc1/scl2 and into scl3/scl1

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: Callek)

References

Details

The current plan is to - put all of the seamonkey linux/windows VMs on the production ESX cluster - put all of the rackable seamonkey hardware in scl3 - put the r2/r3 seamonkey minis in scl1 (c.f. bug 721513 for the net there)
Depends on: 720796
No longer depends on: 720769
No longer depends on: 721512
sea-win32-ix-01 Win 2k3 (32-bit) Physical (iX) 04600 https://inventory.mozilla.org/systems/show/2230/ sea-win32-ix-02 Win 2k3 (32-bit) Physical (iX) 04810 https://inventory.mozilla.org/systems/show/2391/ sea-win32-ix-03 Win 2k3 (32-bit) Physical (iX) 04606 https://inventory.mozilla.org/systems/show/2236/ sea-win32-ix-04 Win 2k3 (32-bit) Physical (iX) 04708 https://inventory.mozilla.org/systems/show/2289/ cb-sea-miniosx64-04 OSX 10.6.2 Physical (r4) 05280 https://inventory.mozilla.org/systems/show/4917/ cb-sea-miniosx64-05 OSX 10.6.2 Physical (r4) 05281 https://inventory.mozilla.org/systems/show/4918/ cb-sea-miniosx64-06 OSX 10.6.2 Physical (r4) 05282 https://inventory.mozilla.org/systems/show/4919/ cb-sea-miniosx64-07 OSX 10.6.2 Physical (r4) 05283 https://inventory.mozilla.org/systems/show/4920/
Status: NEW → ASSIGNED
The iX boxes are now located at SJC1 they have been put into production for Callek and team.
Currently the 13 HP's have been moved to SCL3, the 4 iX boxes are racked and powered at SJC1 on floor 14 rack 102.05 and lastly the 4 mac minis in sonnet chassis are at SCL2 and will be moved to SCL3.
Depends on: 724968
Assignee: mlarrain → dmoore
Bug 730054 tracks the space in scl1 for the r2/r3 minis.
Matt and I just sync'd up on this project. Here's the status: VM's: to be moved to scl3 when ESX is ready cb-seamonkey-linuxmaster-01 cb-sea-linux-tbox cb-seamonkey-linux-01 cb-seamonkey-linux-02 cb-sea-win32-tbox cb-seamonkey-linux64-01 cb-seamonkey-win32-01 cb-seamonkey-win32-02 cb-seamonkey-win32-03 cn-sea-qm-centos5-01 cn-sea-qm-win2k3-01 r2/r3 minis: to be moved to scl1 on the C train on 3/26 blocked on bug 730054 and bug 720553 cb-sea-miniosx01 cb-sea-miniosx02 cb-sea-miniosx64-01 cb-sea-miniosx64-02 cb-sea-miniosx64-03 iX hosts: currently in sjc1 to be moved to scl3 on the C train on 3/26 blocked on bug 720553 sea-win32-01 sea-win32-02 sea-win32-03 sea-win32-04 r4 minis: didn't appear to get moved in bug 724968 not in production to be moved to scl3 on the B train 3/12 I will schedule the move cb-sea-miniosx64-04 cb-sea-miniosx64-05 cb-sea-miniosx64-06 cb-sea-miniosx64-07 HP DL120G7's 13 specified, but 14 moved in bug 724968 no inventory information yet Matt will follow up with dcops to locate and finish these hosts
Assignee: dmoore → mlarrain
Bug 730122 is the move of the r4 minis.
Depends on: 730122
Depends on: 730054
In comment 4, these hosts are irrelevant (in ams1): cn-sea-qm-centos5-01 cn-sea-qm-win2k3-01
(In reply to Dustin J. Mitchell [:dustin] from comment #7) > In comment 4, these hosts are irrelevant (in ams1): > cn-sea-qm-centos5-01 > cn-sea-qm-win2k3-01 Well, irrelevant as far as "needs to move" and easiest to keep out of this context. But I would love to move them to the new ESX cluster and get them out of ams1 if possible :-)
Assignee: mlarrain → dmoore
Depends on: 733962
Callek, I didn't quite make the one-week-warning, but the following hosts will be moving on Monday. Expect roughly a full-day downtime for the windows systems, and possibly more for the minis, depending on how they get crammed into scl1. https://inventory.mozilla.org/systems/show/2230/ - sea-win32-01 https://inventory.mozilla.org/systems/show/2391/ - sea-win32-02 https://inventory.mozilla.org/systems/show/2236/ - sea-win32-03 https://inventory.mozilla.org/systems/show/2289/ - sea-win32-04 https://inventory.mozilla.org/systems/show/1642/ - cb-sea-miniosx01 https://inventory.mozilla.org/systems/show/1632/ - cb-sea-miniosx02 https://inventory.mozilla.org/systems/show/4912/ - cb-sea-miniosx64-01 https://inventory.mozilla.org/systems/show/4913/ - cb-sea-miniosx64-02 https://inventory.mozilla.org/systems/show/1635/ - cb-sea-miniosx64-03
And if you can be around IRC during the day as much as possible, that will be helpful.
The minis will not be moving - space/power isn't ready for them in scl1. So we'll only be moving sea-win32-{01..04}. We'll move the minis in 4 weeks (4/23). I have logins for the windows systems, and Matt is going to work on a powershell to assign them a new IP on reboot, so we may be able to get these up (relatively) hands-free. If not, we'll have someone onsite to configure.
Per Bug 733962 we're only moving the iX ones tonight/tomorrow (In reply to Dustin J. Mitchell [:dustin] from comment #10) > > https://inventory.mozilla.org/systems/show/2230/ - sea-win32-01 > https://inventory.mozilla.org/systems/show/2391/ - sea-win32-02 > https://inventory.mozilla.org/systems/show/2236/ - sea-win32-03 > https://inventory.mozilla.org/systems/show/2289/ - sea-win32-04 All shutdown via a RDP login. Unplug/Move at will.
These four hosts are back up in scl3. However, they can't reach the linux master due to an oversight on my part (not requesting a new network flow). I'll get that fixed up in bug 729504.
DNS should be up now. The flow bug is bug 739504 - sorry for the typo. And it's still open. Arhel tried, but couldn't get the flow to work.
I think this is a good time to revisit the big picture: > VM's: > cb-seamonkey-linuxmaster-01 > cb-sea-linux-tbox > cb-seamonkey-linux-01 > cb-seamonkey-linux-02 > cb-sea-win32-tbox > cb-seamonkey-linux64-01 > cb-seamonkey-win32-01 > cb-seamonkey-win32-02 > cb-seamonkey-win32-03 > cn-sea-qm-centos5-01 > cn-sea-qm-win2k3-01 None are done yet, but we can (and should!) get started on this now. I'll talk to Matt about the process here. > r2/r3 minis: > cb-sea-miniosx01 > cb-sea-miniosx02 > cb-sea-miniosx64-01 > cb-sea-miniosx64-02 > cb-sea-miniosx64-03 stalled on scl1 - bug 730054 > iX hosts: > sea-win32-01 > sea-win32-02 > sea-win32-03 > sea-win32-04 in place in scl3, just waiting on bug 739504. > r4 minis: > cb-sea-miniosx64-04 > cb-sea-miniosx64-05 > cb-sea-miniosx64-06 > cb-sea-miniosx64-07 in place in scl3, imaged, but not yet crash-carted. We can do this soon - maybe Monday when Jake will be there? > HP DL120G7's > 13 specified, but 14 moved in bug 724968 in place in scl3, but not imaged. I don't think we have iLO on these, either, and I know DC ops wanted to do some re-cabling, btu I expect they're finished with that by now. Callek, let's talk about what to do with these - maybe it's time to fire up PuppetAgain, seamonkey-style? Does this block or enable anything else in the list?
(In reply to Dustin J. Mitchell [:dustin] from comment #16) > None are done yet, but we can (and should!) get started on this now. I'll > talk to Matt about the process here. Right, Matt refreshed my memory that because these are parallels, and ESX isn't, the hosts need to be rebuilt. So! With your blessing, I'll get a few new VMs created: sea-master1 sea-vm-linux32-1 sea-vm-linux32-2 sea-vm-linux32-3 sea-vm-linux32-4 sea-vm-linux64-1 sea-vm-win32-1 sea-vm-win32-2 sea-vm-win32-3 sea-vm-win32-4 sea-vm-win32-5 (based on counting the win32, linux, and linux64 hosts in the VM list above) Let me know if the names are OK, and if I should just copy the specs from the existing hosts or change those in some way.
(In reply to Dustin J. Mitchell [:dustin] from comment #16) > I think this is a good time to revisit the big picture: > > > VM's: > > cb-seamonkey-linuxmaster-01 > > cb-sea-linux-tbox This is actually non a parallels VM > > cb-seamonkey-linux-01 > > cb-seamonkey-linux-02 On parallels (before the recent crash) and stopped due to issues was a linux-03 as well, if possible lets plan for that to be turned on as well. > > cb-sea-win32-tbox This is actually a non parallels VM as well > > cn-sea-qm-centos5-01 > > cn-sea-qm-win2k3-01 These are not going away just yet, since they are in ams1, but as we know I'd like to move them to scl3's clusted as well if possible! ----- (In reply to Dustin J. Mitchell [:dustin] from comment #17) > (In reply to Dustin J. Mitchell [:dustin] from comment #16) > > None are done yet, but we can (and should!) get started on this now. I'll > > talk to Matt about the process here. > > Right, Matt refreshed my memory that because these are parallels, and ESX > isn't, the hosts need to be rebuilt. So! With your blessing, I'll get a > few new VMs created: Sure, if its possible to snapshot-copy VMs once I bring one type up we can plan on that instead of necessarily needing all VM space partitioned. > sea-master1 > sea-vm-linux32-1 > sea-vm-linux32-2 > sea-vm-linux32-3 > sea-vm-linux32-4 If we account for the additional VM I mentioned above that was turned off even on the "before major crash" we'd have 5 here. Lets plan for 6 total linux32 and cut into our win vm pool for the extras please. (Since we have 4 iX machines to handle MSVC2010 that takes care of a chunk of our win support) > sea-vm-linux64-1 > sea-vm-win32-1 > sea-vm-win32-2 > sea-vm-win32-3 > sea-vm-win32-4 > sea-vm-win32-5 > > (based on counting the win32, linux, and linux64 hosts in the VM list above) > > Let me know if the names are OK, and if I should just copy the specs from > the existing hosts or change those in some way. I don't have a strong need for the names to be different, in fact consistency is always better :-)
(In reply to Justin Wood (:Callek) from comment #18) > I don't have a strong need for the names to be different, in fact > consistency is always better :-) Oooo just to add, the master I'd like to keep the cb-seamonkey-linuxmaster-01 if even as a CNAME to keep my awesomebar memory alive, if it is not too much trouble, I don't need the whole host called that to be clear :-)
As Callek mentioned that the *-tbox and cn-sea-qm-* ones are ESX already and not parallels, but it would be awesome if we could bring them over to a new consistent setup in the same move. :) Same for getting back the Linux VM we deactivated for Parallels weirdness, as he mentioned. That said, as you seem to be consistently (re)naming those machines in that move, maybe we should think about doing the same for the minis and have one single naming scheme for everything. :)
OK, I got the VM story pretty wrong :) Based on the above, that should be sea-master1 fresh install, manually set up CNAME preserved sea-vm-linux32-1 migrated from cb-sea-linux-tbox (ESX) in sjc1 eventually cloned from sea-vm-linux32-2 once it's ready sea-vm-linux32-2 fresh install, manually set up sea-vm-linux32-3 sea-vm-linux32-4 sea-vm-linux32-5 sea-vm-linux32-6 cloned from sea-vm-linux32-2 once it's ready sea-vm-linux64-1 fresh install, manually set up sea-vm-win32-1 migrated from cb-sea-win32-tbox (ESX) in sjc1 sea-vm-win32-2 sea-vm-win32-3 sea-vm-win32-4 cloned from sea-vm-win32-1 Included by reference: - r2/r3 minis from comment 16 - no rename here; these will die soon - r4 minis from comment 16 - with rename before going into production - HP's from comment 16 - last step on iX windows in comment 16 - replacement of ams1 hosts with local equivalents - from comment 9 (after scl3)
sea-vm-linux32-6 and sea-vm-win32-4 will be filled in by migrating from ams1, so not immediately. We'll clone sea-vm-linux32-2 from sea-vm-linux32-1 like the rest - easier all around. Crash-carting and renaming the minis to be more consistent is in bug 724968. Jake or maybe DC Ops will do that. Getting the HP's set up is in bug 740633. I'll do that.
(In reply to Dustin J. Mitchell [:dustin] from comment #17) > sea-vm-linux32-1 Nit: Should we just keep "*-0n" for numbering, just in case?
Depends on: 740633
Outside of releng, Mozilla is standardizing on non-zero-padded hostnames.
The five r2/r3 minis in comment 16 are scheduled to go to scl1 on May 7.
The four iX windows systems (sea-win32-{01..04}) now have OOB IPs assigned, but we can't configure those without bringing them down and having someone onsite. Since hands are tight and there's lots of other seamonkey stuff to do, let's leave that as a TODO item for after the scl3 move.
(In reply to Dustin J. Mitchell [:dustin] from comment #26) > The four iX windows systems (sea-win32-{01..04}) now have OOB IPs assigned, > but we can't configure those without bringing them down and having someone > onsite. Since hands are tight and there's lots of other seamonkey stuff to > do, let's leave that as a TODO item for after the scl3 move. Sounds good to me, assuming OOB isn't required for anything funky [afaik its not] Though if in the near term, if anyone is caught in that datacenter twiddling their thumbs, and wants to do this, you can ping me on IRC and I can probably bring all 4 systems down with almost no notice to get this done. Not that I expect *anyone* in the near future to be twiddling their thumbs while inside scl3 ;-)
Blocks: 740613
Big-picture refresh: VM's: will be set up in bug 740613 r2/r3 minis: bug 730054 has an ETA, and the hosts are on the 4/24 train (comment 25 is bogus) iX hosts: in production (or ready to be anyway) r4 minis: in place, ready for production HP DL120G7's: all 14 are in place, but iLO needs work - bug 744298 other than the mini moves on 4/24, the only looming deadline here is to get the VMs up and running by the time cb-vmware01 and cb-parallels01 need to move - train I, May 15, is the last chance.
(In reply to Dustin J. Mitchell [:dustin] from comment #28) > Big-picture refresh: > ... > other than the mini moves on 4/24, the only looming deadline here is to get > the VMs up and running by the time cb-vmware01 and cb-parallels01 need to > move - train I, May 15, is the last chance. To be clear, I am HOPING for a temp-storage of parallels until I can get all the new VMs ready. [I thought that was already planned]. The VMs ontop of parallels can't migrate. I doubt even if the VMs were ready tomorrow that I'd have _everything_ from parallels migrated by may 15 (I could be proven wrong though)
cb-parallels01 will definitely move, but let's shoot to get it as cleared-out as possible. I realize that's on us right now :)
The r2/r3's are moving Tuesday: > r2/r3 minis: > cb-sea-miniosx01 > cb-sea-miniosx02 > cb-sea-miniosx64-01 > cb-sea-miniosx64-02 > cb-sea-miniosx64-03 Here's their IP information: cb-sea-miniosx01 IN A 10.12.20.32 cb-sea-miniosx02 IN A 10.12.20.33 cb-sea-miniosx64-01 IN A 10.12.20.34 cb-sea-miniosx64-02 IN A 10.12.20.35 cb-sea-miniosx64-03 IN A 10.12.20.36 netmask 255.255.255.0 gateway 10.12.20.1 I have login information for these, so I'll plan to login and set them up Tuesday morning, before the move. We'll have someone onsite to crash-cart anyway.
Somewhat annoyingly, but as a decent solution to an ugly situation, these minis will be accessed with a different jumphost: jump1.comm-build.scl1.mozilla.com. It is configured identically to the other jumphost, so your access should be just fine. As a reminder, these minis aren't long for the world. We should plan on about one more year with them -- at that point, the datacenter they're in will be evacuated, and we'll have nowhere to put them -- if they're still running by that point. If the hardware dies before that, we really have very little hope of repair or replacement, although of course we'll do what we can.
From Erica: ---- Dustin, The following two servers will not move today, as they cannot be located: 16 No location in inventory Internap Apple - Mac Mini Dustin No tag YM937C2Z9G5 cb-sea-miniosx64-01 No tag 16 No location in inventory Internap Apple - Mac Mini Dustin No tag YM937C3Q9G5 Cb-sea-miniosx64-02 No tag ---- I'll see what I can do.
cb-sea-miniosx64-01 (63.245.210.45) at 00:26:B0:DE:AD:D0 [ether] on eth0 0026.b0de.add0 asx103-06A port 23 cb-sea-miniosx64-02 (63.245.210.46) at 00:26:B0:DE:7E:FC [ether] on eth0 0026.b0de.7efc asx103-06A port 22
So everything but cb-sea-miniosx02 is up now. I'm guessing this is a physical problem - loose cord or bumped power switch - so there's nothing I can do from here. I also don't have remote console on the switch, so I can't get any visibility there. I'll ask Erica to swing by in the morning and investigate. Sorry about that -- that will teach me to have dinner!
DC Ops visited scl1 this morning but left before this host was up. I'll keep working to get someone there.
Thanks to Matt's tender touch, this the last host is back up. Walk softly around it - its power supply was dead.
Big Picture Update: > VM's: > cb-seamonkey-linuxmaster-01 > cb-sea-linux-tbox > cb-seamonkey-linux-01 > cb-seamonkey-linux-02 > cb-sea-win32-tbox > cb-seamonkey-linux64-01 > cb-seamonkey-win32-01 > cb-seamonkey-win32-02 > cb-seamonkey-win32-03 > cn-sea-qm-centos5-01 > cn-sea-qm-win2k3-01 The vmware VMs are in progress, slowly. The parallels VMs will probably migrate onboard cb-parallels01 on the last train, although there's an outside chance we'll have *new* parallels systems in scl3 so we can migrate them one by one. Stay tuned. > r2/r3 minis: > cb-sea-miniosx01 > cb-sea-miniosx02 > cb-sea-miniosx64-01 > cb-sea-miniosx64-02 > cb-sea-miniosx64-03 Done > iX hosts: > sea-win32-01 > sea-win32-02 > sea-win32-03 > sea-win32-04 Done > r4 minis: > cb-sea-miniosx64-04 > cb-sea-miniosx64-05 > cb-sea-miniosx64-06 > cb-sea-miniosx64-07 Done > HP DL120G7's Still in place in scl3, but not imaged - bug 744298. This isn't an scl3-timeline thing, so I'm not pushing on it.
cb-parallels01 is on the 5/8 train. That's next week!
Blocks: 720796
No longer depends on: 720796
Blocks: 740633
No longer depends on: 740633
bug 740633 is only blocked on dc ops finding time to do it - not on this bug.
No longer blocks: 740633
Parallels VM's: > cb-seamonkey-linuxmaster-01 dmitchell@jump1 ~ $ nc -vz cb-seamonkey-linuxmaster-01.community.scl3.mozilla.com 22 Connection to cb-seamonkey-linuxmaster-01.community.scl3.mozilla.com 22 port [tcp/ssh] succeeded! > cb-seamonkey-linux-01 (/builds had to be fsck -y'd -- it may be wise to reformat this partition) dmitchell@jump1 ~ $ nc -vz cb-seamonkey-linux-01.community.scl3.mozilla.com 22 Connection to cb-seamonkey-linux-01.community.scl3.mozilla.com 22 port [tcp/ssh] succeeded! > cb-seamonkey-linux-02 bad root fs - known > cb-seamonkey-linux64-01 dmitchell@jump1 ~ $ nc -vz cb-seamonkey-linux64-01.community.scl3.mozilla.com 22 Connection to cb-seamonkey-linux64-01.community.scl3.mozilla.com 22 port [tcp/ssh] succeeded! > cb-seamonkey-win32-01 > cb-seamonkey-win32-02 > cb-seamonkey-win32-03 dmitchell@jump1 ~ $ nc -vz cb-seamonkey-win32-01.community.scl3.mozilla.com 3389 Connection to cb-seamonkey-win32-01.community.scl3.mozilla.com 3389 port [tcp/ms-wbt-server] succeeded! dmitchell@jump1 ~ $ nc -vz cb-seamonkey-win32-02.community.scl3.mozilla.com 3389 Connection to cb-seamonkey-win32-02.community.scl3.mozilla.com 3389 port [tcp/ms-wbt-server] succeeded! dmitchell@jump1 ~ $ nc -vz cb-seamonkey-win32-03.community.scl3.mozilla.com 3389 Connection to cb-seamonkey-win32-03.community.scl3.mozilla.com 3389 port [tcp/ms-wbt-server] succeeded! So all that's left here is * Callek to set up SM VMs and get them cloned (bug 740613) * allowing us to kill some of the parallels VMs * HP's (bug 744298)
Assignee: dmoore → dustin
Per Callek's request, I've put cb-seamonkey-linuxmaster-01 (old master) at 63.245.223.11, and moved sea-master1 to 63.245.223.98. We'll need to swap those back when the latter is ready to be the real master. I fixed up the mozilla.org DNS zone to point everything to the new master's IP (where the old master is at the moment): +seamonkey-clobberer IN CNAME sea-master1.community.scl3.mozilla.com. +cb-seamonkey-linuxmaster-01 IN CNAME sea-master1.community.scl3.mozilla.com.
Well, these are transitioned to scl3 and scl1, but the last two comments outline some fixing-up that needs to occur. Both are on Callek atm.
Assignee: dustin → bugspam.Callek
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.