Closed Bug 489940 Opened 16 years ago Closed 16 years ago

Retune Win32 VMs to speed up win32 build times

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Platform:
x86
Windows Server 2003
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: joduinn, Assigned: joduinn)

References

Details

On the win32 VMs, change drive E from: 30GB FAT32 ...to: 80gb NTFS. This needs to be done as an temp interim step until we figure out if bug#472223 does scale or not. Already done are: moz2-win32-slave01 moz2-win32-slave03 moz2-win32-slave06
Priority: -- → P2
Some non-default NTFS settings helped reduce build times a little further, see details here: https://wiki.mozilla.org/ReferencePlatforms/Win32#Add_drive_E
moz2-win32-slave08 now also done.
moz2-win32-slave07 now also done.
Switching to NTFS results in less disk-used per build-tree. For a mozilla-central-win32 directory: Total Objdir .hg rest of source Summed file size 4.79G 4.34G 183M 264M ------------------------------------------------------------ NTFS space on disk 5.06G 4.40G 309M 355M FAT space on disk 6.16G 4.61G 831M 735M ------------------------------------------------------------ Improvment 1.10G 210M 522M 380M so we save 1.1GB for that build tree. As most of that comes from .hg and the rest of the source directory we should make proportionally similar savings on other trees too (even if they have less checkins).
moz2-win32-slave09 done.
moz-win32-slave10 done.
moz-win32-slave11 done.
Completed to date: 01/03/06/07/08/09/10/11
Here are the steps I followed to do this disk upgrade. 1) verify slave has smaller (30gb) driveE, and that the diskarray the VM is on has at least 130gb free. We want to leave approx 100gb free after all is done. 2) on buildbot master, tell slave to "gracefully shutdown". 3) login as admin, right click computer icon, select properties, select disk management, change drive E to drive Z 4) power off VM 5) in VI, right click on VM, select "edit settings" and add additional drive. Note it should be on same disk array as rest of VM, and should be 80gb in size. 6) power on VM 7) login as admin, right click computer icon, select properties, select disk management. Right click new disk, assign to drive E, create "extended partition", and format as NTFS. Use non-default NTFS settings specified in https://wiki.mozilla.org/ReferencePlatforms/Win32#Add_drive_E 8) create e:\builds\moz2_slave\ 9) copy following files from Z to E z:\builds\moz2_slave\info\* z:\builds\moz2_slave\buildbot.tac 10) reboot VM 11) verify that slave did reconnect to master, and that builds are running ok. Note that if this doesnt happen within a min/two, check the buildbot process is able to write to e:\builds\moz2_slave. ...Monitor for a day to make sure builds/tests are ok... 12) on buildbot master, tell slave to "gracefully shutdown". 13) login as admin, power off VM. 14) in VI, right click on VM, select "edit settings" and delete the 30gb disk (drive Z). 15) reboot VM 16) verify that slave did reconnect to master, and that builds are running ok.
Finished moz2-win32-slave14 + slave 26. Completed to date: 01/03/06/07/08/09/10/11/14/26
win32-slave20 (nee moz2-win32-slave20) appears to have both disks, and be running builds on NTFS. Needs finishing off ? Anything else in progress ?
Yes, I need to finish the clean up on win32-slave20. I'll get on that right away, or when it becomes free...
Finished moz2-win32-slave20. Completed to date: 01/03/06/07/08/09/10/11/14/20/26
We stopped doing these disk upgrades because of the corrupt NTFS filesystem woes in bug#496712. Upgrading to mozilla-build1.4 seems to have fixed that. Can alice/myself now continue bumping up disks on the remaining slaves?
(In reply to comment #14) > We stopped doing these disk upgrades because of the corrupt NTFS filesystem > woes in bug#496712. Upgrading to mozilla-build1.4 seems to have fixed that. > > Can alice/myself now continue bumping up disks on the remaining slaves? If you want to be really safe you should wait until we deploy MB1.4 on the moz2-win32-* slaves (bug 505565)
(In reply to comment #15) > (In reply to comment #14) > > We stopped doing these disk upgrades because of the corrupt NTFS filesystem > > woes in bug#496712. Upgrading to mozilla-build1.4 seems to have fixed that. > > > > Can alice/myself now continue bumping up disks on the remaining slaves? > > If you want to be really safe you should wait until we deploy MB1.4 on the > moz2-win32-* slaves (bug 505565) Excellent point!
Depends on: 505565
I think we can finish the rest of the slaves now!
Now getting physical hardware for win2k3 builds, so no need to continue with this VM tweaking. Closing as GOODENOUGHFORNOW.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
I had this comment written a while ago but never submitted it: Update: * moz2-win32-slave04 got done by way of being recloned at some point * everything from win32-slave30 onwards was created with a 80G NTFS disk * the list of remaining slaves with 30G FAT disks is moz2-win32-slave02,05,12,13,15,16,17,18,19,21,22,23,24,25,27,28,29 * aka 17 of the 56 slaves we're running in moz2 Additionally, the following slaves were still have a 30G disk allocated on the storage array which isn't in use: moz2-win32-slave01,06,10,11,14,20,26 When I tried deleting the 30G on moz2-win32-slave01 (by shutting the VM down and editing the VM settings, removing the stray disk) ESX didn't clean up the space on the array. Any ideas about that Phong ?
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.