764693 - Bring up all new SeaMonkey windows VMs in scl3

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Justin Wood (:Callek)]

In Bug 740613 we got VMs ready/setup from old sjc1 VM's/Setups, including spots to move our Parallel Machines. And all this after discussion about final state.

Doing this step will help free us up in some places.

We need to bring up new images for:

sea-vm-win32-1,3,4

And create a snapshot for storage based on the image used.

Based on clone from sea-vm-win32-2

All of sea-vm-win32-{1,2,3,4} are at consecutive IP's and should already have flows setup. (63.245.223.72 -> 63.245.223.75) and are in the .community.scl3.mozilla.com VLAN/Subnet.

This will require coordinated downtime of sea-vm-win32-2, which includes deleting in-flux data on E:\ (which auto-regenerates for builds) and removing severe secrets like ssh keys, and buildbot.tac. Prior to copying this image.

After these are up, we can additionally delete some windows VMs in use for SeaMonkey, and free up space/CPU for other systems. (this will need at least a few hours lead time to prepare for shutting them down cleanly)

Can be Deleted (and good riddance):
* cb-seamonkey-win32-01
* cb-seamonkey-win32-02
* cb-seamonkey-win32-03

Lastly we had planned to bring down the ams1 VM we have as well with these VMs, however I'd request we keep that one around pending the completion (and sanity verification) of Bug 761596.

I am very flexible on the downtime date, but would prefer no earlier than this Friday afternoon.

Comment 1

[Justin Wood (:Callek)]

OK, so ETA in [starting to] bringing these up, is tomorrow at 10:30 PT

To summarize actions for sanity.

Action Items for :Callek (done as of now):
 * Clean VM of .ssh keys
 * Clean VM of bulk in E:\builds\
 * Clean VM of temp files in C:\*
 * Removed password from E:\builds\slave\buildbot.tac and renamed file so buildbot won't startup

Action Items for IT (:Aj):
 * Create a backup of VM, for storage somewhere [on fs2?]
 * Delete sea-vm-win2k3-1 image
 * Populate sea-vm-win2k3-{1,3,4} with sea-vm-win2k3-2
 ** Consecutive IPs [including *-2] at (63.245.223.72 -> 63.245.223.75)
 * Fix hostname and IP on these tree VMs
 * Reboot These VMs
 * Give :Callek the "all clear" (here or in IRC)

--> At this point bug can either be closed, or kept open for Part 2 of this work.

Action Items for SeaMonkey-Releng (ewong/callek):
 * For EACH slave
 **    Put back .ssh keys/known_hosts/etc. on all 4 VMs
 **    Enter correct slave password, and hostname into E:\builds\slave\buildbot.tac.bak and rename to buildbot.tac
 **    Verify hostname & domain of slave is correct
 **    test ssh connection to: stage, symbolpush, aus2
 **    Double check IP address (and hostmask) is set properly
 **    Restart VM
 * Verify in buildbot UI that the VM's connected and have started taking jobs
 * Monitor through to thursday, midday that jobs are good.
 * If they are (on cb-seamonkey-win32-*)
 **    Disable/delete buildbot.tac on and await jobs to finish.
 **    Delete .ssh directories from all users
 **    Delete temp files from all users
 **    Power Off these hosts
 **    Note here, or file a new bug (depending on if this bug is closed) that IT can delete these slaves.

Action Items for IT for Part 2:
 * On cb-parallels01
 **   Power off and delete space used by cb-seamonkey-win32-*
 **   Delete backups in fs2 of the parallel images for cb-seamonkey-win32-* [yes I'm sure] :-)

Comment 2

[Adrian J Fernandez [:Aj]]

:Callek trying to reach you via IRC to clear some things up.

Also want to make sure it's ok to shutdown sea-vm-win32-2 for the clone but might end up cloning it while it's still up (prefer to have it down).

Comment 3

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #1)
>  * Populate sea-vm-win2k3-{1,3,4} with sea-vm-win2k3-2
>  ** Consecutive IPs [including *-2] at (63.245.223.72 -> 63.245.223.75)

So I was pointed out that I did a typo here:
<Aj>	sure thing, no problem, also just to want to double clarify that you do want them renamed from sea-vm-win32-{1-4} to sea-vm-win2k3-{1-4} , correct ?
<Aj>	yea cause in the bug it mentions that

So to be clear (in bug too) we want them named |sea-vm-win32-{1-4}| as current, the latter was just a mistype by me.

Comment 4

[Adrian J Fernandez [:Aj]]

Following was done;
* sea-vm-win32-1 deleted
* sea-vm-win32-2 made to a template; sea-vm-win2k3-2
* sea-vm-win32-{1-4} currently being deployed (cloned) from sea-vm-win2k3-2 template.

Will update bug once cloning completes and IPs have been updated to the correct ones.

Comment 5

[Adrian J Fernandez [:Aj]]

sea-vm-win32-{1-4}.community.scl3.mozilla.com are online.

:Callek notified via irc as well.

Would be best to create a separate bug for the remaining items (for consistency's sake).

If there's anything wrong or that needs to be changed, please let me know.

Comment 6

[Edmund Wong (:ewong)]

(In reply to Adrian Fernandez [:Aj] from comment #5)
> sea-vm-win32-{1-4}.community.scl3.mozilla.com are online.
> 
> :Callek notified via irc as well.
> 
> Would be best to create a separate bug for the remaining items (for
> consistency's sake).
> 

Bug #766502 is the continuation of this bug but for the SeaMonkey Releng part.
Leaving this bug open for part II.

Comment 7

[Adrian J Fernandez [:Aj]]

* cb-seamonkey-win32-01
* cb-seamonkey-win32-02
* cb-seamonkey-win32-03

Deleted as requested.

As for part II, the requested VMs were deleted in Bug 765134