780022 - [tracker] reimage 73 linux32, linux64 ix builders as win64 builders

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In meeting w/Melissa this morning, and now w/hwine, we couldnt find a bug for this despite prior discussions/meetings, so filing now for tracking.)

Once we offload production linux desktop builds to AWS, (bug#772446), please reimage the linux32, linux64 ix builders as win64 builders. Increasing our win64 pool like this should help improve wait times on our win32, win64 builds.

Filing in RelEng for now, while we figure out logistics of using new slaves on release trains, and how many physical linux32,64 builders can be spared for helping win64.

Note: For the ix machines in 650castro, bug#712456 tracks upgrading the physical hardware as part of moving these machines from 650castro to a "real" colo.

Comment 1

[Amy Rich [:arr] [:arich]]

Converting the hardware that's already in scl1 is easier.  Those machines include:

linux-ix-slave01.build.scl1
linux-ix-slave02.build.scl1
linux-ix-slave03.build.scl1
linux-ix-slave04.build.scl1
linux-ix-slave05.build.scl1
linux-ix-slave06.build.scl1
linux-ix-slave07.build.scl1
linux-ix-slave08.build.scl1
linux-ix-slave09.build.scl1
linux-ix-slave10.build.scl1
linux-ix-slave11.build.scl1
linux-ix-slave12.build.scl1
linux-ix-slave13.build.scl1
linux-ix-slave14.build.scl1
linux-ix-slave15.build.scl1
linux-ix-slave16.build.scl1
linux-ix-slave17.build.scl1
linux-ix-slave18.build.scl1
linux-ix-slave19.build.scl1
linux-ix-slave20.build.scl1
linux-ix-slave21.build.scl1
linux-ix-slave22.build.scl1
linux-ix-slave23.build.scl1
linux-ix-slave24.build.scl1
linux-ix-slave25.build.scl1
linux-ix-slave26.build.scl1
linux-ix-slave27.build.scl1
linux-ix-slave28.build.scl1
linux-ix-slave29.build.scl1
linux-ix-slave30.build.scl1
linux-ix-slave31.build.scl1
linux-ix-slave32.build.scl1
linux-ix-slave33.build.scl1
linux-ix-slave34.build.scl1
linux-ix-slave35.build.scl1
linux-ix-slave36.build.scl1
linux-ix-slave37.build.scl1
linux-ix-slave38.build.scl1
linux-ix-slave39.build.scl1
linux-ix-slave40.build.scl1
linux-ix-slave41.build.scl1
linux-ix-slave42.build.scl1

linux64-ix-slave01.build.scl1
linux64-ix-slave02.build.scl1
linux64-ix-slave03.build.scl1
linux64-ix-slave04.build.scl1
linux64-ix-slave05.build.scl1
linux64-ix-slave06.build.scl1
linux64-ix-slave07.build.scl1
linux64-ix-slave08.build.scl1
linux64-ix-slave09.build.scl1
linux64-ix-slave10.build.scl1
linux64-ix-slave11.build.scl1
linux64-ix-slave12.build.scl1
linux64-ix-slave13.build.scl1
linux64-ix-slave14.build.scl1
linux64-ix-slave15.build.scl1
linux64-ix-slave16.build.scl1
linux64-ix-slave17.build.scl1
linux64-ix-slave18.build.scl1
linux64-ix-slave19.build.scl1
linux64-ix-slave20.build.scl1
linux64-ix-slave21.build.scl1
linux64-ix-slave22.build.scl1
linux64-ix-slave23.build.scl1
linux64-ix-slave24.build.scl1
linux64-ix-slave25.build.scl1
linux64-ix-slave26.build.scl1
linux64-ix-slave27.build.scl1
linux64-ix-slave28.build.scl1
linux64-ix-slave29.build.scl1
linux64-ix-slave30.build.scl1
linux64-ix-slave31.build.scl1
linux64-ix-slave32.build.scl1
linux64-ix-slave33.build.scl1
linux64-ix-slave34.build.scl1
linux64-ix-slave35.build.scl1
linux64-ix-slave36.build.scl1
linux64-ix-slave38.build.scl1
linux64-ix-slave39.build.scl1
linux64-ix-slave40.build.scl1
linux64-ix-slave41.build.scl1


There are additional machines in mtv1 that need hardware upgrades to move to a datacenter, so that means additional time and effort, and we would need to find space for them in scl3 (how much space we have available in scl3 depends on how many w8 test boxes we purchase):

mv-moz2-linux-ix-slave01.build.mtv1
mv-moz2-linux-ix-slave02.build.mtv1
mv-moz2-linux-ix-slave03.build.mtv1
mv-moz2-linux-ix-slave04.build.mtv1
mv-moz2-linux-ix-slave05.build.mtv1
mv-moz2-linux-ix-slave06.build.mtv1
mv-moz2-linux-ix-slave07.build.mtv1
mv-moz2-linux-ix-slave08.build.mtv1
mv-moz2-linux-ix-slave09.build.mtv1
mv-moz2-linux-ix-slave10.build.mtv1
mv-moz2-linux-ix-slave11.build.mtv1
mv-moz2-linux-ix-slave12.build.mtv1
mv-moz2-linux-ix-slave13.build.mtv1
mv-moz2-linux-ix-slave14.build.mtv1
mv-moz2-linux-ix-slave15.build.mtv1
mv-moz2-linux-ix-slave16.build.mtv1
mv-moz2-linux-ix-slave17.build.mtv1
mv-moz2-linux-ix-slave18.build.mtv1
mv-moz2-linux-ix-slave19.build.mtv1
mv-moz2-linux-ix-slave20.build.mtv1
mv-moz2-linux-ix-slave21.build.mtv1
mv-moz2-linux-ix-slave22.build.mtv1
mv-moz2-linux-ix-slave23.build.mtv1

The remaining ix machines in mtv1 are mw32 machines and covered under a different bug.

Comment 2

[Hal Wine [:hwine] use NI!]

Moved the mv-moz2-linux machines to bug 784721, and the upgrade dependency with them.

Comment 3

[Hal Wine [:hwine] use NI!]

We will be moving all of the linux 32 & 64 bit boxes to windows 64 eventually. We will add batches that can be moved to this bug as they become available.

If there is any lead time needed before first move, please start that work if it's one shot (as done in bug 758275 comment #18)

Also, let us know a rough estimate of the turn around time on the reimaging and associated infrastructure changes per batch.

Thanks!

Comment 4

[Chris Cooper [:coop] (he/him)]

Based on current build/try waittimes, we can safely re-image the following machines to w64 now:

Linux (build slaves)
linux-ix-slave38.build.scl1
linux-ix-slave39.build.scl1
linux-ix-slave40.build.scl1
linux-ix-slave41.build.scl1
linux-ix-slave42.build.scl1

Linux64 (trybuild slaves)
linux64-ix-slave36.build.scl1
linux64-ix-slave38.build.scl1
linux64-ix-slave39.build.scl1
linux64-ix-slave40.build.scl1
linux64-ix-slave41.build.scl1

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

VLANs are changed to vlan40.

Comment 6

[Matthew Larrain[:MaRu]]

Added these to DHCP in Windows will started the reimaging process shortly

Comment 7

[Amy Rich [:arr] [:arich]]

(In reply to Chris Cooper [:coop] from comment #4)

These machines have been reimaged as w64.

Comment 8

[Amy Rich [:arr] [:arich]]

hwine, coop, joduinn: are these all of the machines we'll be switching?  If so, I'll close out this bug.

Comment 9

[Hal Wine [:hwine] use NI!]

:arr - no, eventually all in comment #1 will be converted over time. Sounds like the first batch is done.

To avoid confusion, let do future batches in dependent bugs, and keep this one as the tracker for the full batch.

Comment 10

[Amy Rich [:arr] [:arich]]

Sounds good, I'll reassign this one to you as a tracker and you can open up specific bugs in the relops queue for batches of machines for us to reimage.

Comment 11

[Hal Wine [:hwine] use NI!]

updated summary - started with 83 in this bug (42 of lin32, 41 of lin64) from comment #1

10 in production with close of bug 786035

Next step: RelEng to identify next batch to reimage

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Some of these machines could go ahead, the ones on try where we dropped non-mock builds already. See
 http://buildbot-master31.srv.releng.scl3.mozilla.com:8101/buildslaves
for slaves with 'no builders' against them. Bug 804766 for *-vmw-*, I don't see a bug for the mac slaves.

Comment 13

[Armen [:armenzg]]

What is the current status of this bug? Thanks!

Comment 14

[Amy Rich [:arr] [:arich]]

The linux 32 and 64 machines were converted to foopies and mock builders.  We have a handful (6 of each) left that will be decommissioned when they are no longer serving their current purpose.  I don't think there's any further action left for this.

Comment 15

[Armen [:armenzg]]

Are these the ones that are staying around a little longer? (I want to adjust slavealloc)
linux-ix-slave01.build.scl1.mozilla.com has address 10.12.48.195
linux-ix-slave02.build.scl1.mozilla.com has address 10.12.48.196
linux-ix-slave03.build.scl1.mozilla.com has address 10.12.48.197
linux-ix-slave04.build.scl1.mozilla.com has address 10.12.48.198
linux-ix-slave05.build.scl1.mozilla.com has address 10.12.48.199
linux-ix-slave06.build.scl1.mozilla.com has address 10.12.48.200
linux64-ix-slave01.build.scl1.mozilla.com has address 10.12.49.44
linux64-ix-slave02.build.scl1.mozilla.com has address 10.12.49.45
linux64-ix-slave03.build.scl1.mozilla.com has address 10.12.49.46
linux64-ix-slave04.build.scl1.mozilla.com has address 10.12.49.47
linux64-ix-slave05.build.scl1.mozilla.com has address 10.12.49.48
linux64-ix-slave06.build.scl1.mozilla.com has address 10.12.49.49

It seems that these have been decommissioned:
linux-ix-slave[07-31]
mv-moz2-linux-ix-slave[01-23]

According to the esr cycle [1] we would need to keep those machines around until Firefox 25 ships which is in 2013-10-29 [1][2]

Another alternative is if we had *vmw-* VMs (like we used to have) we could re-purpose those machines.

If we're fine to wait then we can wait.

On another, why do we have esr17 nightly builds?

[1] http://mozorg.cdn.mozilla.net/media/img/firefox/organizations/release-overview.png
[2] https://wiki.mozilla.org/RapidRelease/Calendar

Comment 16

[Chris AtLee [:catlee]]

Yes, we need to keep those machines for ESR17. We do nightlies on ESR so our partners can validate the builds before do the final release builds.

Comment 17

[Hal Wine [:hwine] use NI!]

nothing more to do here