1226180 - decommission t-yosemite-r5 machines

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

We need to do rolling decommissions of the t-yosemite-r5 machines so we have enough space for the new r7 machines.

Comment 1

[Amy Rich [:arr] [:arich]]

To make life easier for dcops, I'm going to pick machines all from the same rack that are (mostly) sharing chassis together. There's one or two outliers that are sharing with an OS X build machine.

Here are the first 48 machines we're going to decommission (also listed in https://docs.google.com/spreadsheets/d/1o4C9aUDmyIwn7VgAxur2_GIlOwx7f92hii_oL8lnjMU/edit#gid=72561714):

t-yosemite-r5-0008.test.releng.scl3.mozilla.com
t-yosemite-r5-0009.test.releng.scl3.mozilla.com
t-yosemite-r5-0010.test.releng.scl3.mozilla.com
t-yosemite-r5-0011.test.releng.scl3.mozilla.com
t-yosemite-r5-0051.test.releng.scl3.mozilla.com
t-yosemite-r5-0054.test.releng.scl3.mozilla.com
t-yosemite-r5-0055.test.releng.scl3.mozilla.com
t-yosemite-r5-0056.test.releng.scl3.mozilla.com
t-yosemite-r5-0057.test.releng.scl3.mozilla.com
t-yosemite-r5-0058.test.releng.scl3.mozilla.com
t-yosemite-r5-0059.test.releng.scl3.mozilla.com
t-yosemite-r5-0060.test.releng.scl3.mozilla.com
t-yosemite-r5-0061.test.releng.scl3.mozilla.com
t-yosemite-r5-0062.test.releng.scl3.mozilla.com
t-yosemite-r5-0063.test.releng.scl3.mozilla.com
t-yosemite-r5-0064.test.releng.scl3.mozilla.com
t-yosemite-r5-0065.test.releng.scl3.mozilla.com
t-yosemite-r5-0066.test.releng.scl3.mozilla.com
t-yosemite-r5-0067.test.releng.scl3.mozilla.com
t-yosemite-r5-0068.test.releng.scl3.mozilla.com
t-yosemite-r5-0069.test.releng.scl3.mozilla.com
t-yosemite-r5-0070.test.releng.scl3.mozilla.com
t-yosemite-r5-0071.test.releng.scl3.mozilla.com
t-yosemite-r5-0072.test.releng.scl3.mozilla.com
t-yosemite-r5-0073.test.releng.scl3.mozilla.com
t-yosemite-r5-0074.test.releng.scl3.mozilla.com
t-yosemite-r5-0075.test.releng.scl3.mozilla.com
t-yosemite-r5-0076.test.releng.scl3.mozilla.com
t-yosemite-r5-0077.test.releng.scl3.mozilla.com
t-yosemite-r5-0078.test.releng.scl3.mozilla.com
t-yosemite-r5-0079.test.releng.scl3.mozilla.com
t-yosemite-r5-0080.test.releng.scl3.mozilla.com
t-yosemite-r5-0081.test.releng.scl3.mozilla.com
t-yosemite-r5-0082.test.releng.scl3.mozilla.com
t-yosemite-r5-0083.test.releng.scl3.mozilla.com
t-yosemite-r5-0084.test.releng.scl3.mozilla.com
t-yosemite-r5-0085.test.releng.scl3.mozilla.com
t-yosemite-r5-0086.test.releng.scl3.mozilla.com
t-yosemite-r5-0087.test.releng.scl3.mozilla.com
t-yosemite-r5-0088.test.releng.scl3.mozilla.com
t-yosemite-r5-0089.test.releng.scl3.mozilla.com
t-yosemite-r5-0090.test.releng.scl3.mozilla.com
t-yosemite-r5-0091.test.releng.scl3.mozilla.com
t-yosemite-r5-0093.test.releng.scl3.mozilla.com
t-yosemite-r5-0104.test.releng.scl3.mozilla.com
t-yosemite-r5-0105.test.releng.scl3.mozilla.com
t-yosemite-r5-0106.test.releng.scl3.mozilla.com
t-yosemite-r5-0107.test.releng.scl3.mozilla.com

Coop, can you please disable them and create patches to pull them out of configs (unless you want to do one big config cleanup at the end)?

Dcops: can you please plan out which of the new machines are going where and put the PDU, switch, rack, etc info in https://docs.google.com/spreadsheets/d/1o4C9aUDmyIwn7VgAxur2_GIlOwx7f92hii_oL8lnjMU/edit#gid=1373260922 ? We're starting at t-yosemite-r7-0065.test.releng.scl3.mozilla.com since we've already racked and installed 0001 - 0064. 

Once that's information is in place, I'll update inventory with the new host's info. And modify nagios.

Comment 2

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1226180_production_config.py.patch

Attached you can find the production_config.py

Comment 3

[Kim Moir [:kmoir] ET]

Comment on attachment 8689583 [details] [diff] [review]
bug1226180_production_config.py.patch

You need to disable the machines in slavealloc before we can land this change.  Also, we will need patches to remove the machines from slavealloc once the existing jobs have completed.

Comment 4

[Vlad Ciobancai [:vladC] [:vciobancai]]

Disable them in slavealloc and added "decommed" as a note to them

Comment 5

[Van Le [:van]]

do you guys plan to format these machines as well?

Comment 6

[Kim Moir [:kmoir] ET]

No we are not going to reuse them.

Comment 7

[Amy Rich [:arr] [:arich]]

(In reply to Van Le [:van] from comment #5)

I believe the last we talked, we could just hand them over to the disposal company with a chain of custody that verified they disposed of the disk in an approved way, right Van?

Comment 8

[Van Le [:van]]

:arr, yup we can do that during the disposal process. i wasn't sure if you guys were going to donate them like last time so a pre-wipe would be required.

Comment 9

[Amy Rich [:arr] [:arich]]

I think the decision was that donation was too time consuming for everyone involved.

Comment 10

[Amy Rich [:arr] [:arich]]

I've commented out this batch of machines from nagios so that we don't get alerts for them as buildbot stops.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Merged to production at http://hg.mozilla.org/build/buildbot-configs/rev/61e319b19e0b

Comment 12

[Kim Moir [:kmoir] ET]

arr: do you know when these machines will be decommissioned and new ones moved into their racks?  We're seeing high pending counts on the yosemite pools so we are anxious to add the new machines

Comment 13

[Kim Moir [:kmoir] ET]

I enabled 65-86 because I saw they were puppetized, they are now running jobs.

Comment 14

[Amy Rich [:arr] [:arich]]

Dcops has been replacing the 48 r5 minis in the first batch yesterday and today. They're turning them on in batches of 10 to install.

Comment 15

[Amy Rich [:arr] [:arich]]

Kim is decommissioning the following machines today so dcops can work on staging their replacements (and installing them on Friday):


t-yosemite-r5-0012.test.releng.scl3.mozilla.com
t-yosemite-r5-0013.test.releng.scl3.mozilla.com
t-yosemite-r5-0014.test.releng.scl3.mozilla.com
t-yosemite-r5-0015.test.releng.scl3.mozilla.com
t-yosemite-r5-0016.test.releng.scl3.mozilla.com
t-yosemite-r5-0017.test.releng.scl3.mozilla.com
t-yosemite-r5-0018.test.releng.scl3.mozilla.com
t-yosemite-r5-0019.test.releng.scl3.mozilla.com
t-yosemite-r5-0020.test.releng.scl3.mozilla.com
t-yosemite-r5-0021.test.releng.scl3.mozilla.com
t-yosemite-r5-0022.test.releng.scl3.mozilla.com
t-yosemite-r5-0023.test.releng.scl3.mozilla.com
t-yosemite-r5-0024.test.releng.scl3.mozilla.com
t-yosemite-r5-0025.test.releng.scl3.mozilla.com
t-yosemite-r5-0026.test.releng.scl3.mozilla.com
t-yosemite-r5-0027.test.releng.scl3.mozilla.com
t-yosemite-r5-0028.test.releng.scl3.mozilla.com
t-yosemite-r5-0052.test.releng.scl3.mozilla.com
t-yosemite-r5-0053.test.releng.scl3.mozilla.com
t-yosemite-r5-0029.test.releng.scl3.mozilla.com
t-yosemite-r5-0030.test.releng.scl3.mozilla.com
t-yosemite-r5-0031.test.releng.scl3.mozilla.com
t-yosemite-r5-0032.test.releng.scl3.mozilla.com
t-yosemite-r5-0033.test.releng.scl3.mozilla.com

Comment 16

[Kim Moir [:kmoir] ET]

They are disabled in slavealloc, just waiting for the jobs to finish.  Van said he would pull them after PT lunch

Comment 17

[Amy Rich [:arr] [:arich]]

The following hosts have been decommissioned:

t-yosemite-r5-0034.test.releng.scl3.mozilla.com
t-yosemite-r5-0035.test.releng.scl3.mozilla.com
t-yosemite-r5-0036.test.releng.scl3.mozilla.com
t-yosemite-r5-0037.test.releng.scl3.mozilla.com
t-yosemite-r5-0038.test.releng.scl3.mozilla.com
t-yosemite-r5-0003.test.releng.scl3.mozilla.com
t-yosemite-r5-0039.test.releng.scl3.mozilla.com
t-yosemite-r5-0040.test.releng.scl3.mozilla.com
t-yosemite-r5-0095.test.releng.scl3.mozilla.com
t-yosemite-r5-0096.test.releng.scl3.mozilla.com
t-yosemite-r5-0097.test.releng.scl3.mozilla.com
t-yosemite-r5-0098.test.releng.scl3.mozilla.com
t-yosemite-r5-0099.test.releng.scl3.mozilla.com
t-yosemite-r5-0100.test.releng.scl3.mozilla.com
t-yosemite-r5-0101.test.releng.scl3.mozilla.com
t-yosemite-r5-0102.test.releng.scl3.mozilla.com
t-yosemite-r5-0103.test.releng.scl3.mozilla.com
t-yosemite-r5-0094.test.releng.scl3.mozilla.com

Comment 18

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1226180_production_config.py_v2.patch

Attached you can find the production_config.py updated by removing the yosemite_r5 that has been decomm

Comment 19

[Vlad Ciobancai [:vladC] [:vciobancai]]

Comment on attachment 8707330 [details] [diff] [review]
bug1226180_production_config.py_v2.patch

I wanted to ask you for review not feedback

Comment 20

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/b05f8de5ec9d

Comment 21

[Amy Rich [:arr] [:arich]]

We're aiming for 2016-02-09 to do the hardware work for batch 5.

Comment 22

[Amy Rich [:arr] [:arich]]

The following hosts have been decommissioned:

t-snow-r4-0147.test.releng.scl3.mozilla.com
t-snow-r4-0148.test.releng.scl3.mozilla.com
t-snow-r4-0149.test.releng.scl3.mozilla.com
t-snow-r4-0150.test.releng.scl3.mozilla.com
t-snow-r4-0151.test.releng.scl3.mozilla.com
t-snow-r4-0152.test.releng.scl3.mozilla.com
t-snow-r4-0154.test.releng.scl3.mozilla.com
t-snow-r4-0155.test.releng.scl3.mozilla.com
t-snow-r4-0156.test.releng.scl3.mozilla.com
t-snow-r4-0157.test.releng.scl3.mozilla.com
t-snow-r4-0158.test.releng.scl3.mozilla.com
t-snow-r4-0159.test.releng.scl3.mozilla.com
t-snow-r4-0161.test.releng.scl3.mozilla.com
t-snow-r4-0162.test.releng.scl3.mozilla.com
t-snow-r4-0163.test.releng.scl3.mozilla.com
t-yosemite-r5-0041.test.releng.scl3.mozilla.com
t-yosemite-r5-0043.test.releng.scl3.mozilla.com
t-yosemite-r5-0044.test.releng.scl3.mozilla.com
t-yosemite-r5-0045.test.releng.scl3.mozilla.com
t-yosemite-r5-0046.test.releng.scl3.mozilla.com
t-yosemite-r5-0047.test.releng.scl3.mozilla.com
t-yosemite-r5-0048.test.releng.scl3.mozilla.com
t-yosemite-r5-0049.test.releng.scl3.mozilla.com
t-yosemite-r5-0050.test.releng.scl3.mozilla.com
t-yosemite-r5-0006.test.releng.scl3.mozilla.com
t-yosemite-r5-0005.test.releng.scl3.mozilla.com
t-yosemite-r5-0004.test.releng.scl3.mozilla.com

Comment 23

[Kim Moir [:kmoir] ET]

Attachment: bug1226180.patch

Comment 24

[Van Le [:van]]

batch5 swap and reimage completed.

Comment 25

[Van Le [:van]]

:arr do you have the hostnames for the new mac minis in batch5?

Comment 26

[Kim Moir [:kmoir] ET]

I thinks the hostnames are covered in bug 1247372

Comment 27

[Kim Moir [:kmoir] ET]

latest patch in production

Comment 28

[Phil Ringnalda (:philor)]

Could probably do 0001, 0002 and 0007 now, apparently it's been 117 days since there was last any work for them.

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Phil Ringnalda (:philor) from comment #28)
> Could probably do 0001, 0002 and 0007 now, apparently it's been 117 days
> since there was last any work for them.

Yes please!

Comment 30

[Alin Selagea [:aselagea]]

I did some DB cleaning for the t-yosemite-r5 machines. 
From what I could see, 001, 002 and 007 are still online with "status=production" in inventory.

Comment 31

[Amy Rich [:arr] [:arich]]

:alin: are those ready to be decommissioned now?

Comment 32

[Alin Selagea [:aselagea]]

Yes, they are.

Comment 33

[Amy Rich [:arr] [:arich]]

All references to t-yosemite-r5 removed from nagios:
   efbe01f..c581bf1  master -> master

1, 2, and 7 shut down and decommissioned in inventory.
Van, please unrack these for disposal/donation.

Comment 34

[Van Le [:van]]

1,2, and 7 unracked. will wipe/pull drive and dispose.