849022 - [mtv1->scl3] upgrade and move win32-ix-ref, linux-ix-ref and linux64-ix-ref

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Chris Cooper [:coop] (he/him)]

Lots of blockers remain here, but getting this one filed. These boxes also need heatsink/RAM upgrades, similar to what happened to the other iX machines in bug 774829.

The machines in question are:
mw32-ix-slave01
mw32-ix-slave02
mw32-ix-slave03
mw32-ix-slave04
mw32-ix-slave05
mw32-ix-slave06
mw32-ix-slave07
mw32-ix-slave08
mw32-ix-slave09
mw32-ix-slave10
mw32-ix-slave11
mw32-ix-slave12

Comment 1

[Amy Rich [:arr] [:arich]]

DIscussion on whether or not we keep these once they're done with their current purpose since they are out of warranty.

Comment 2

[Amy Rich [:arr] [:arich]]

These machines are out of warranty, and likely we will be decommissioning them instead of repurposing them.

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #0)
> mw32-ix-slave07
> mw32-ix-slave09
> mw32-ix-slave10

These 3 slaves are already disabled and could be decomm-ed immediately if necessary. Also happy for relops to wait and do them all as a batch.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From meeting w/RelEng and IT, the list of machines at question is: 


linux-ix-slave01.build.scl1.mozilla.com
linux-ix-slave02.build.scl1.mozilla.com
linux-ix-slave03.build.scl1.mozilla.com
linux-ix-slave04.build.scl1.mozilla.com
linux-ix-slave05.build.scl1.mozilla.com
linux-ix-slave06.build.scl1.mozilla.com
linux-ix-ref.build.mtv1.mozilla.com

linux64-ix-slave01.build.scl1.mozilla.com
linux64-ix-slave02.build.scl1.mozilla.com
linux64-ix-slave03.build.scl1.mozilla.com
linux64-ix-slave04.build.scl1.mozilla.com
linux64-ix-slave05.build.scl1.mozilla.com
linux64-ix-slave06.build.scl1.mozilla.com
linux64-ix-ref.build.scl1.mozilla.com

mw32-ix-slave01.build.scl1.mozilla.com
mw32-ix-slave02.build.scl1.mozilla.com
mw32-ix-slave03.build.scl1.mozilla.com
mw32-ix-slave04.build.scl1.mozilla.com
mw32-ix-slave05.build.scl1.mozilla.com
mw32-ix-slave06.build.scl1.mozilla.com
mw32-ix-slave07.build.scl1.mozilla.com [*]
mw32-ix-slave08.build.scl1.mozilla.com
mw32-ix-slave09.build.scl1.mozilla.com [*]
mw32-ix-slave10.build.scl1.mozilla.com [*]
mw32-ix-slave11.build.scl1.mozilla.com
mw32-ix-slave12.build.scl1.mozilla.com
win32-ix-ref.build.mtv1.mozilla.com


[*] note that from comment#3, mw32-ix-slave07/09/10 are already disabled at this time.

Comment 5

[hwine]

These will all go to SCL3 to be used.

 - A number of these machines need to be physically upgraded before moving to SCL3. With coordination, these upgrades can be done prior to move.

 - Some of these machines may be "released early" from ESR 17 duty. Those will be immediately re-purposed (and moved to SCL3 if need be for usage).

Bugs will be separately filed for each of these steps.

Comment 6

[Armen [:armenzg]]

hwine, would you be able to drive this to completion?

Comment 7

[Armen [:armenzg]]

All of these machines have been disabled.
Please proceed with the required hardware upgrades required to move them to scl3.

The following machines linux-ix-slave0{4,5} and linux64-ix-slave0{1,2} were re-purposed as w64-ix-slave[159-162]. I have not put them yet on production. We can go ahead and move them to scl3.

Thank you very much for your help!

(In reply to John O'Duinn [:joduinn] from comment #4)
> From meeting w/RelEng and IT, the list of machines at question is: 
> 
> 
> linux-ix-slave01.build.scl1.mozilla.com
> linux-ix-slave02.build.scl1.mozilla.com
> linux-ix-slave03.build.scl1.mozilla.com
> linux-ix-slave04.build.scl1.mozilla.com
> linux-ix-slave05.build.scl1.mozilla.com
> linux-ix-slave06.build.scl1.mozilla.com
> linux-ix-ref.build.mtv1.mozilla.com
> 
> linux64-ix-slave01.build.scl1.mozilla.com
> linux64-ix-slave02.build.scl1.mozilla.com
> linux64-ix-slave03.build.scl1.mozilla.com
> linux64-ix-slave04.build.scl1.mozilla.com
> linux64-ix-slave05.build.scl1.mozilla.com
> linux64-ix-slave06.build.scl1.mozilla.com
> linux64-ix-ref.build.scl1.mozilla.com
> 
> mw32-ix-slave01.build.scl1.mozilla.com
> mw32-ix-slave02.build.scl1.mozilla.com
> mw32-ix-slave03.build.scl1.mozilla.com
> mw32-ix-slave04.build.scl1.mozilla.com
> mw32-ix-slave05.build.scl1.mozilla.com
> mw32-ix-slave06.build.scl1.mozilla.com
> mw32-ix-slave07.build.scl1.mozilla.com
> mw32-ix-slave08.build.scl1.mozilla.com
> mw32-ix-slave09.build.scl1.mozilla.com
> mw32-ix-slave10.build.scl1.mozilla.com
> mw32-ix-slave11.build.scl1.mozilla.com
> mw32-ix-slave12.build.scl1.mozilla.com
> win32-ix-ref.build.mtv1.mozilla.com
>

Comment 8

[hwine]

moving to dcops for hardware upgrades and physical moves

Comment 9

[Van Le [:van]]

to confirm, you'd like us to move these 3 hosts to scl3?

win32-ix-ref.build.mtv1.mozilla.com
linux64-ix-ref.build.scl1.mozilla.com
linux-ix-ref.build.mtv1.mozilla.com

Comment 10

[Armen [:armenzg]]

(In reply to Van Le [:van] from comment #9)
> to confirm, you'd like us to move these 3 hosts to scl3?
> 
> win32-ix-ref.build.mtv1.mozilla.com
> linux64-ix-ref.build.scl1.mozilla.com
> linux-ix-ref.build.mtv1.mozilla.com

That makes sense. Good catch!
For some reason I thought the whole group of machines was on mtv1.

Comment 11

[Amy Rich [:arr] [:arich]]

This bug has gotten way too confusing since it talks about different sets of machines with different purposes with different groups that need to act on them.  I'm going to close it in favor of reopening the one that was already filed for the purpose of upgrading the machines in MTV1, and then open a new one to repurpose the machines in scl1.