Decommissioning vmware win32 builders

RESOLVED FIXED

Status

RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: mlarrain, Assigned: phong)

Tracking

Details

(Reporter)

Description

7 years ago
Per requests for speed increases in Vmware and the fact that the builders are failing left and right I am going to be powering off the win32 builders for a week and then pulling them off to dracula and backing them up to tape.
We only need a backup of the template - although it's probably better to leave that template around and not delete it.  The builders themselves can be reconstructed from that template.
Group: mozilla-corporation-confidential
Duplicate of this bug: 673021
Since the 22nd, we've had 478 builds on the ix machines, and 135 on the VMs.
For the record, this is still waiting for a "go" from catlee, after the try/build machines are rebalanced.
wait times look ok so far. Let's power off 50% of the VMs and see where we're at.
(Reporter)

Comment 6

7 years ago
Chris do you have a list oh which ones I can power off?
Assignee: server-ops-releng → mlarrain
We're going to be doing try-w32-slave24..36, win32-slave36..59. I've disabled them all in slavealloc.

try-w32-slave24..36 and win32-slave50..59 are ready to be shut off now. Need to wait for the others to finish up builds.
(Reporter)

Comment 8

7 years ago
Thanks catlee I will be servicing this request later today.
(Reporter)

Comment 9

7 years ago
catlee is the win32-slave50-59 the moz2-win32-slave50-59
(Reporter)

Comment 10

7 years ago
Per irc convo with catlee

 <catlee> yeah, moz2-win32-XX == win32-XX
Hosts removed from nagios.
Once we're done with this, these hosts should be removed from inventory, dns, and dhcp as well.
win32-slave60 can go, too - it's in preprod.
(Reporter)

Comment 14

7 years ago
ok arr can you kill nagios and ill kill the host
we'll need to remove these from slavealloc when we're sure we don't need them any more.
(Reporter)

Comment 16

7 years ago
Powered off try-w32-slave24..36 and win32-slave50..60 in vmware and took care of nagios with arr's help. Waiting for next set to come out of prod to power those off next.
The rest are ready to be shut off.
(In reply to comment #17)
> The rest are ready to be shut off.

to be more specific, win32-slave36..49 are ready to be shut off
(Reporter)

Comment 19

7 years ago
Disabled nagios alerts and powered off VM's. Next step is to balance the VM's across the hosts.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 20

7 years ago
Didn't mean to close this forgot there were more windows VM's to power off.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 21

7 years ago
These are the machines left to get turned off. Releng can you guys give me some insight on when I can do that

moz2-win32-slave01
moz2-win32-slave04
moz2-win32-slave06
moz2-win32-slave07
moz2-win32-slave08
moz2-win32-slave09
moz2-win32-slave10
moz2-win32-slave11
moz2-win32-slave20
moz2-win32-slave21
moz2-win32-slave02
moz2-win32-slave03
moz2-win32-slave26
moz2-win32-slave30
moz2-win32-slave31
moz2-win32-slave32
moz2-win32-slave33
moz2-win32-slave34
moz2-win32-slave35
(Reporter)

Comment 22

7 years ago
Looks like i forgot to check the try machines here is that list as well. 

try-w32-slave01-23
I'm disabling the rest of the try slaves today; will let you know when those are ready to shut off.
try-w32-slave01-23 are ready to be turned off
try-w32-slave01-23 have been removed from nagios
try-w32-slave01-23 have been turned off.
We still have 19 moz2-win32-slaveXX (where XX is in the range 01-35) vms in service.  Will we be shutting those down as well, or are they slated to stay active?
(Reporter)

Comment 28

7 years ago
I am waiting to hear from Catlee for when I can remove the systems from Nagios and shut them down.
Status: REOPENED → ASSIGNED
(Reporter)

Comment 29

7 years ago
Per IRC convo with catlee I am powering off moz2-win32-slave01-35
(Reporter)

Comment 30

7 years ago
Removed from nagios and powered off.
per meeting with IT yesterday:

* all VMs now powered off.
(Reporter)

Updated

7 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
I think this was resolved a bit early - these VMs still exist, and are on storage that we need to get rid of.  Can we delete them? Coop?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Phong: nuke at will; close when finished.
Assignee: mlarrain → phong
(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> I think this was resolved a bit early - these VMs still exist, and are on
> storage that we need to get rid of.  Can we delete them? Coop?

Yes, they can be deleted.
(Reporter)

Comment 35

7 years ago
YAY!!! I can't wait to destroy them doing it tomorrow morning :)
I didn't see them in vmware, so maybe phong has already deleted them. I removed them from DNS/dhcp/inventory
Phong confirms he deleted the actual vms.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.