673174 - Decommissioning vmware win32 builders

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Matthew Larrain[:MaRu]]

Per requests for speed increases in Vmware and the fact that the builders are failing left and right I am going to be powering off the win32 builders for a week and then pulling them off to dracula and backing them up to tape.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

We only need a backup of the template - although it's probably better to leave that template around and not delete it.  The builders themselves can be reconstructed from that template.

Comment 3

[Chris AtLee [:catlee]]

Since the 22nd, we've had 478 builds on the ix machines, and 135 on the VMs.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

For the record, this is still waiting for a "go" from catlee, after the try/build machines are rebalanced.

Comment 5

[Chris AtLee [:catlee]]

wait times look ok so far. Let's power off 50% of the VMs and see where we're at.

Comment 6

[Matthew Larrain[:MaRu]]

Chris do you have a list oh which ones I can power off?

Comment 7

[Chris AtLee [:catlee]]

We're going to be doing try-w32-slave24..36, win32-slave36..59. I've disabled them all in slavealloc.

try-w32-slave24..36 and win32-slave50..59 are ready to be shut off now. Need to wait for the others to finish up builds.

Comment 8

[Matthew Larrain[:MaRu]]

Thanks catlee I will be servicing this request later today.

Comment 9

[Matthew Larrain[:MaRu]]

catlee is the win32-slave50-59 the moz2-win32-slave50-59

Comment 10

[Matthew Larrain[:MaRu]]

Per irc convo with catlee

 <catlee> yeah, moz2-win32-XX == win32-XX

Comment 11

[Amy Rich [:arr] [:arich]]

Hosts removed from nagios.

Comment 12

[Amy Rich [:arr] [:arich]]

Once we're done with this, these hosts should be removed from inventory, dns, and dhcp as well.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

win32-slave60 can go, too - it's in preprod.

Comment 14

[Matthew Larrain[:MaRu]]

ok arr can you kill nagios and ill kill the host

Comment 15

[Chris AtLee [:catlee]]

we'll need to remove these from slavealloc when we're sure we don't need them any more.

Comment 16

[Matthew Larrain[:MaRu]]

Powered off try-w32-slave24..36 and win32-slave50..60 in vmware and took care of nagios with arr's help. Waiting for next set to come out of prod to power those off next.

Comment 17

[Chris AtLee [:catlee]]

The rest are ready to be shut off.

Comment 18

[Chris AtLee [:catlee]]

(In reply to comment #17)
> The rest are ready to be shut off.

to be more specific, win32-slave36..49 are ready to be shut off

Comment 19

[Matthew Larrain[:MaRu]]

Disabled nagios alerts and powered off VM's. Next step is to balance the VM's across the hosts.

Comment 20

[Matthew Larrain[:MaRu]]

Didn't mean to close this forgot there were more windows VM's to power off.

Comment 21

[Matthew Larrain[:MaRu]]

These are the machines left to get turned off. Releng can you guys give me some insight on when I can do that

moz2-win32-slave01
moz2-win32-slave04
moz2-win32-slave06
moz2-win32-slave07
moz2-win32-slave08
moz2-win32-slave09
moz2-win32-slave10
moz2-win32-slave11
moz2-win32-slave20
moz2-win32-slave21
moz2-win32-slave02
moz2-win32-slave03
moz2-win32-slave26
moz2-win32-slave30
moz2-win32-slave31
moz2-win32-slave32
moz2-win32-slave33
moz2-win32-slave34
moz2-win32-slave35

Comment 22

[Matthew Larrain[:MaRu]]

Looks like i forgot to check the try machines here is that list as well. 

try-w32-slave01-23

Comment 23

[Chris AtLee [:catlee]]

I'm disabling the rest of the try slaves today; will let you know when those are ready to shut off.

Comment 24

[Chris AtLee [:catlee]]

try-w32-slave01-23 are ready to be turned off

Comment 25

[Amy Rich [:arr] [:arich]]

try-w32-slave01-23 have been removed from nagios

Comment 26

[Amy Rich [:arr] [:arich]]

try-w32-slave01-23 have been turned off.

Comment 27

[Amy Rich [:arr] [:arich]]

We still have 19 moz2-win32-slaveXX (where XX is in the range 01-35) vms in service.  Will we be shutting those down as well, or are they slated to stay active?

Comment 28

[Matthew Larrain[:MaRu]]

I am waiting to hear from Catlee for when I can remove the systems from Nagios and shut them down.

Comment 29

[Matthew Larrain[:MaRu]]

Per IRC convo with catlee I am powering off moz2-win32-slave01-35

Comment 30

[Matthew Larrain[:MaRu]]

Removed from nagios and powered off.

Comment 31

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting with IT yesterday:

* all VMs now powered off.

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

I think this was resolved a bit early - these VMs still exist, and are on storage that we need to get rid of.  Can we delete them? Coop?

Comment 33

[Dustin J. Mitchell [:dustin] (he/him)]

Phong: nuke at will; close when finished.

Comment 34

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #32)
> I think this was resolved a bit early - these VMs still exist, and are on
> storage that we need to get rid of.  Can we delete them? Coop?

Yes, they can be deleted.

Comment 35

[Matthew Larrain[:MaRu]]

YAY!!! I can't wait to destroy them doing it tomorrow morning :)

Comment 36

[Amy Rich [:arr] [:arich]]

I didn't see them in vmware, so maybe phong has already deleted them. I removed them from DNS/dhcp/inventory

Comment 37

[Amy Rich [:arr] [:arich]]

Phong confirms he deleted the actual vms.