Last Comment Bug 780022 - [tracker] reimage 73 linux32, linux64 ix builders as win64 builders
: [tracker] reimage 73 linux32, linux64 ix builders as win64 builders
Status: RESOLVED FIXED
[tracker] [reit]
:
Product: Release Engineering
Classification: Other
Component: Buildduty (show other bugs)
: other
: x86 Mac OS X
: -- normal (vote)
: ---
Assigned To: Hal Wine [:hwine] (use NI)
: Armen Zambrano [:armenzg] (EDT/UTC-4)
Mentors:
Depends on: 772446 784850 786035 795496
Blocks: 784891 863236
  Show dependency treegraph
 
Reported: 2012-08-02 16:07 PDT by John O'Duinn [:joduinn] (please use "needinfo?" flag)
Modified: 2014-10-29 07:04 PDT (History)
9 users (show)
See Also:
Crash Signature:
(edit)
Machine State: ---
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description John O'Duinn [:joduinn] (please use "needinfo?" flag) 2012-08-02 16:07:01 PDT
(In meeting w/Melissa this morning, and now w/hwine, we couldnt find a bug for this despite prior discussions/meetings, so filing now for tracking.)

Once we offload production linux desktop builds to AWS, (bug#772446), please reimage the linux32, linux64 ix builders as win64 builders. Increasing our win64 pool like this should help improve wait times on our win32, win64 builds.

Filing in RelEng for now, while we figure out logistics of using new slaves on release trains, and how many physical linux32,64 builders can be spared for helping win64.

Note: For the ix machines in 650castro, bug#712456 tracks upgrading the physical hardware as part of moving these machines from 650castro to a "real" colo.
Comment 1 Amy Rich [:arr] [:arich] 2012-08-21 11:01:15 PDT
Converting the hardware that's already in scl1 is easier.  Those machines include:

linux-ix-slave01.build.scl1
linux-ix-slave02.build.scl1
linux-ix-slave03.build.scl1
linux-ix-slave04.build.scl1
linux-ix-slave05.build.scl1
linux-ix-slave06.build.scl1
linux-ix-slave07.build.scl1
linux-ix-slave08.build.scl1
linux-ix-slave09.build.scl1
linux-ix-slave10.build.scl1
linux-ix-slave11.build.scl1
linux-ix-slave12.build.scl1
linux-ix-slave13.build.scl1
linux-ix-slave14.build.scl1
linux-ix-slave15.build.scl1
linux-ix-slave16.build.scl1
linux-ix-slave17.build.scl1
linux-ix-slave18.build.scl1
linux-ix-slave19.build.scl1
linux-ix-slave20.build.scl1
linux-ix-slave21.build.scl1
linux-ix-slave22.build.scl1
linux-ix-slave23.build.scl1
linux-ix-slave24.build.scl1
linux-ix-slave25.build.scl1
linux-ix-slave26.build.scl1
linux-ix-slave27.build.scl1
linux-ix-slave28.build.scl1
linux-ix-slave29.build.scl1
linux-ix-slave30.build.scl1
linux-ix-slave31.build.scl1
linux-ix-slave32.build.scl1
linux-ix-slave33.build.scl1
linux-ix-slave34.build.scl1
linux-ix-slave35.build.scl1
linux-ix-slave36.build.scl1
linux-ix-slave37.build.scl1
linux-ix-slave38.build.scl1
linux-ix-slave39.build.scl1
linux-ix-slave40.build.scl1
linux-ix-slave41.build.scl1
linux-ix-slave42.build.scl1

linux64-ix-slave01.build.scl1
linux64-ix-slave02.build.scl1
linux64-ix-slave03.build.scl1
linux64-ix-slave04.build.scl1
linux64-ix-slave05.build.scl1
linux64-ix-slave06.build.scl1
linux64-ix-slave07.build.scl1
linux64-ix-slave08.build.scl1
linux64-ix-slave09.build.scl1
linux64-ix-slave10.build.scl1
linux64-ix-slave11.build.scl1
linux64-ix-slave12.build.scl1
linux64-ix-slave13.build.scl1
linux64-ix-slave14.build.scl1
linux64-ix-slave15.build.scl1
linux64-ix-slave16.build.scl1
linux64-ix-slave17.build.scl1
linux64-ix-slave18.build.scl1
linux64-ix-slave19.build.scl1
linux64-ix-slave20.build.scl1
linux64-ix-slave21.build.scl1
linux64-ix-slave22.build.scl1
linux64-ix-slave23.build.scl1
linux64-ix-slave24.build.scl1
linux64-ix-slave25.build.scl1
linux64-ix-slave26.build.scl1
linux64-ix-slave27.build.scl1
linux64-ix-slave28.build.scl1
linux64-ix-slave29.build.scl1
linux64-ix-slave30.build.scl1
linux64-ix-slave31.build.scl1
linux64-ix-slave32.build.scl1
linux64-ix-slave33.build.scl1
linux64-ix-slave34.build.scl1
linux64-ix-slave35.build.scl1
linux64-ix-slave36.build.scl1
linux64-ix-slave38.build.scl1
linux64-ix-slave39.build.scl1
linux64-ix-slave40.build.scl1
linux64-ix-slave41.build.scl1


There are additional machines in mtv1 that need hardware upgrades to move to a datacenter, so that means additional time and effort, and we would need to find space for them in scl3 (how much space we have available in scl3 depends on how many w8 test boxes we purchase):

mv-moz2-linux-ix-slave01.build.mtv1
mv-moz2-linux-ix-slave02.build.mtv1
mv-moz2-linux-ix-slave03.build.mtv1
mv-moz2-linux-ix-slave04.build.mtv1
mv-moz2-linux-ix-slave05.build.mtv1
mv-moz2-linux-ix-slave06.build.mtv1
mv-moz2-linux-ix-slave07.build.mtv1
mv-moz2-linux-ix-slave08.build.mtv1
mv-moz2-linux-ix-slave09.build.mtv1
mv-moz2-linux-ix-slave10.build.mtv1
mv-moz2-linux-ix-slave11.build.mtv1
mv-moz2-linux-ix-slave12.build.mtv1
mv-moz2-linux-ix-slave13.build.mtv1
mv-moz2-linux-ix-slave14.build.mtv1
mv-moz2-linux-ix-slave15.build.mtv1
mv-moz2-linux-ix-slave16.build.mtv1
mv-moz2-linux-ix-slave17.build.mtv1
mv-moz2-linux-ix-slave18.build.mtv1
mv-moz2-linux-ix-slave19.build.mtv1
mv-moz2-linux-ix-slave20.build.mtv1
mv-moz2-linux-ix-slave21.build.mtv1
mv-moz2-linux-ix-slave22.build.mtv1
mv-moz2-linux-ix-slave23.build.mtv1

The remaining ix machines in mtv1 are mw32 machines and covered under a different bug.
Comment 2 Hal Wine [:hwine] (use NI) 2012-08-22 10:43:55 PDT
Moved the mv-moz2-linux machines to bug 784721, and the upgrade dependency with them.
Comment 3 Hal Wine [:hwine] (use NI) 2012-08-22 11:04:18 PDT
We will be moving all of the linux 32 & 64 bit boxes to windows 64 eventually. We will add batches that can be moved to this bug as they become available.

If there is any lead time needed before first move, please start that work if it's one shot (as done in bug 758275 comment #18)

Also, let us know a rough estimate of the turn around time on the reimaging and associated infrastructure changes per batch.

Thanks!
Comment 4 Chris Cooper [:coop] [away until Aug 29] 2012-08-22 11:58:49 PDT
Based on current build/try waittimes, we can safely re-image the following machines to w64 now:

Linux (build slaves)
linux-ix-slave38.build.scl1
linux-ix-slave39.build.scl1
linux-ix-slave40.build.scl1
linux-ix-slave41.build.scl1
linux-ix-slave42.build.scl1

Linux64 (trybuild slaves)
linux64-ix-slave36.build.scl1
linux64-ix-slave38.build.scl1
linux64-ix-slave39.build.scl1
linux64-ix-slave40.build.scl1
linux64-ix-slave41.build.scl1
Comment 5 Dustin J. Mitchell [:dustin] 2012-08-22 13:58:44 PDT
VLANs are changed to vlan40.
Comment 6 Matthew Larrain[:MaRu] 2012-08-22 16:12:31 PDT
Added these to DHCP in Windows will started the reimaging process shortly
Comment 7 Amy Rich [:arr] [:arich] 2012-08-25 18:20:51 PDT
(In reply to Chris Cooper [:coop] from comment #4)

These machines have been reimaged as w64.
Comment 8 Amy Rich [:arr] [:arich] 2012-08-27 09:02:28 PDT
hwine, coop, joduinn: are these all of the machines we'll be switching?  If so, I'll close out this bug.
Comment 9 Hal Wine [:hwine] (use NI) 2012-08-27 12:57:48 PDT
:arr - no, eventually all in comment #1 will be converted over time. Sounds like the first batch is done.

To avoid confusion, let do future batches in dependent bugs, and keep this one as the tracker for the full batch.
Comment 10 Amy Rich [:arr] [:arich] 2012-08-27 13:01:06 PDT
Sounds good, I'll reassign this one to you as a tracker and you can open up specific bugs in the relops queue for batches of machines for us to reimage.
Comment 11 Hal Wine [:hwine] (use NI) 2012-08-27 21:31:21 PDT
updated summary - started with 83 in this bug (42 of lin32, 41 of lin64) from comment #1

10 in production with close of bug 786035

Next step: RelEng to identify next batch to reimage
Comment 12 Nick Thomas [:nthomas] 2012-10-23 16:00:02 PDT
Some of these machines could go ahead, the ones on try where we dropped non-mock builds already. See
 http://buildbot-master31.srv.releng.scl3.mozilla.com:8101/buildslaves
for slaves with 'no builders' against them. Bug 804766 for *-vmw-*, I don't see a bug for the mac slaves.
Comment 13 Armen Zambrano [:armenzg] (EDT/UTC-4) 2013-04-03 08:53:46 PDT
What is the current status of this bug? Thanks!
Comment 14 Amy Rich [:arr] [:arich] 2013-04-03 08:56:09 PDT
The linux 32 and 64 machines were converted to foopies and mock builders.  We have a handful (6 of each) left that will be decommissioned when they are no longer serving their current purpose.  I don't think there's any further action left for this.
Comment 15 Armen Zambrano [:armenzg] (EDT/UTC-4) 2013-04-18 06:11:57 PDT
Are these the ones that are staying around a little longer? (I want to adjust slavealloc)
linux-ix-slave01.build.scl1.mozilla.com has address 10.12.48.195
linux-ix-slave02.build.scl1.mozilla.com has address 10.12.48.196
linux-ix-slave03.build.scl1.mozilla.com has address 10.12.48.197
linux-ix-slave04.build.scl1.mozilla.com has address 10.12.48.198
linux-ix-slave05.build.scl1.mozilla.com has address 10.12.48.199
linux-ix-slave06.build.scl1.mozilla.com has address 10.12.48.200
linux64-ix-slave01.build.scl1.mozilla.com has address 10.12.49.44
linux64-ix-slave02.build.scl1.mozilla.com has address 10.12.49.45
linux64-ix-slave03.build.scl1.mozilla.com has address 10.12.49.46
linux64-ix-slave04.build.scl1.mozilla.com has address 10.12.49.47
linux64-ix-slave05.build.scl1.mozilla.com has address 10.12.49.48
linux64-ix-slave06.build.scl1.mozilla.com has address 10.12.49.49

It seems that these have been decommissioned:
linux-ix-slave[07-31]
mv-moz2-linux-ix-slave[01-23]

According to the esr cycle [1] we would need to keep those machines around until Firefox 25 ships which is in 2013-10-29 [1][2]

Another alternative is if we had *vmw-* VMs (like we used to have) we could re-purpose those machines.

If we're fine to wait then we can wait.

On another, why do we have esr17 nightly builds?

[1] http://mozorg.cdn.mozilla.net/media/img/firefox/organizations/release-overview.png
[2] https://wiki.mozilla.org/RapidRelease/Calendar
Comment 16 Chris AtLee [:catlee] 2013-04-18 06:38:46 PDT
Yes, we need to keep those machines for ESR17. We do nightlies on ESR so our partners can validate the builds before do the final release builds.
Comment 17 Hal Wine [:hwine] (use NI) 2013-07-03 16:44:47 PDT
nothing more to do here

Note You need to log in before you can comment on or make changes to this bug.