This is a tracking bug for the W64 project for automating the installing of applications and systems changes.
Attempting to build an MSI package of VS2010 to see if that fixes the install issue we are running into with this.
Summary: Automation for W64 project → created MDT-driven automated install configuration for windows builders
Was able to deploy Visual Studio 2010 via MDT. Moving on to finishing the last updates needed for this image and we should be able to start deploying with it shortly.
Last night I was able to push the hard drive improvements per the wiki using a command like script of; fsutil behavior set disable8dot3 1 && fsutil behavior set disablelastaccess 1 Able to change the pagefile size via the command line and am tweaking it to run via automation using; wmic.exe computersystem set AutomaticManagedPagefile=False && wmic.exe pagefileset set InitialSize=4048,MaximumSize=4048 Setting the power options via command line requires getting the GUID from the system using powercfg after that the cli script looks like; powercfg -setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c
mozilla tools installs correctly now
Got the pagefile to actually work ran into an issue where it didn't want to run from MDT itself so I wrote a batch script that combines pagefile and powercfg into one. Working on getting the ssh keys into the right place. Per conversation with dustin we are just moving them from a folder on the MDT box and copying them across to the .ssh folder.
I'd like to hear a definitive answer on whether we should be installing production ssh keys with MDT. The share on MDT is password authenticated, so it's trustworthy.
(correct dependency - not win8 related) We'll get an answer on the keys. The current keys installed are old, and not the latest production ones. Installing the fresh keys is the major step RelEng does prior to enabling the device
I got an answer from coop, actually - we should install the staging keys, and releng can do the prod keys manually.
Writing up process list for discussion on what to use MDT for vs what to use GPO for. Meeting scheduled with Dustin for tomorrow to review and update documentation. This is for both builders and testers.
Assignee: mlarrain → server-ops-releng
Whiteboard: [reit] [tracker] → [reit] [tracker] [2013Q1]
Bug 827497 is the last stretch of getting the MDT task sequence that starts with a capture to actually work.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
This bug is actually for Throop, not Halsted and should therefore remain open as a tracker until Throop is complete.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [reit] [tracker] [2013Q1] → [reit] [tracker] [2013Q2]
Summary: created MDT-driven automated install configuration for windows builders → created MDT-driven automated install configuration for windows 2008R2 (w64) builders
Started with the Server 2008 R2 clean which had been set up by our consultant. Replaced the OS with a version of 2008 R2 that included Service Pack 1. The Task sequence was set up according the info located at: https://wiki.mozilla.org/ReferencePlatforms/Win64 The following are the individual items from that page and how they were addressed: Administrator password The administrator password is original set during the creation of the task sequence in MDT. It is then later controlled by GPO , User_Local_Builder. This GPO was created by copying the User_local_tester users data and adding cltbld user to the local administrator group. That was done to do to various items that is needed to be done by this user that is locked down to the administrator group on Server 2008. Virtual Memory Un-check "Automatically manage paging file size for all drives" Custom size: 4048-4048 This was handled by the “Paging File Size” step in the build task Sequence. Reg file located at: %SCRIPTROOT%\RegEdits\2008memPage.reg" Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management] "ClearPageFileAtShutdown"=dword:00000000 "DisablePagingExecutive"=dword:00000000 "LargeSystemCache"=dword:00000001 "NonPagedPoolQuota"=dword:00000000 "NonPagedPoolSize"=dword:00000000 "PagedPoolQuota"=dword:00000000 "PagedPoolSize"=dword:00000000 "SecondLevelDataCache"=dword:00000000 "SessionPoolSize"=dword:00000004 "SessionViewSize"=dword:00000030 "SystemPages"=dword:00000000 "PagingFiles"=hex(7):63,00,3a,00,5c,00,70,00,61,00,67,00,65,00,66,00,69,00,6c,\ 00,65,00,2e,00,73,00,79,00,73,00,20,00,34,00,30,00,34,00,38,00,20,00,34,00,\ 30,00,34,00,38,00,00,00,00,00 "PhysicalAddressExtension"=dword:00000001 "ExistingPageFiles"=hex(7):5c,00,3f,00,3f,00,5c,00,43,00,3a,00,5c,00,70,00,61,\ 00,67,00,65,00,66,00,69,00,6c,00,65,00,2e,00,73,00,79,00,73,00,00,00,00,00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters] "BootId"=dword:00000008 "BaseTime"=dword:16c177f2 Hard-drive adjustments Based off of bug :https://bugzilla.mozilla.org/show_bug.cgi?id=798639 and a conversation with Releng it was decided, due to the available disk space on the current hardware, that the builders will only configured with a single drive, the C: drive. Clean Manager Clean Manger is installed using the servermanagercmd in the “Enable Clean Manger” build step: cmd /c servermanagercmd -install Desktop-Experience -resultPath results.xml Daemon tools Based in a conversation with Releng, Daemon Tools was only needed to manually set up a builder. It was used to mount the Visual Studio ISOs. However, if Daemon Tools is need in the future. It is now an available application in MDT. Create cltbld user User creation was copied over from previous task sequences, “Create cltbld account” Build step. It runs the following command : net user cltbld xxxPASSxxx /add Basic OS Setup Verify the name of the NetBios computer name and the DNS suffix are correct This handled by the current naming script that is used with all the MDT installs. Check for updates Check for updates a second time after the first batch of updates are installed Windows Server 2008 R2 Service Pack 1 x64 Edition (KB976932) This was handled by the OS version that was imported and used for the task sequences. Disable automatic updates This is handled by the “Disable_auto_updates” GPO. Which under: Computer Configuration> Policies> Administrative Templates > Windows Components /Windows Update, configure automatic updates is disabled. Keep Windows Firewall ON and add exceptions once you install the SSH and VNC servers This handled by the build steps using the netsh cmd: “Allow sshtunnel through firewall” netsh advfirewall firewall add rule name="Allow SSH tunnel" dir=in action=allow program="C:\slave\test\build\bin\ssltunnel.exe" “Allow VNC through firewall” netsh advfirewall firewall add rule name="Allow VNC" dir=in action=allow program="C:\Program Files\uvnc bvba\ultravnc\winvnc.exe" A firewall acceptation was set for Python as well: “Allow python through firewall” netsh advfirewall firewall add rule name="Allow Python" dir=in action=allow program="C:\mozilla-build\buildbotve\scripts\python.exe" The firewall is also managed through GPO, “Firewall_exceptions_testers_builders”. This was done so that firewalls setting may be managed without re-imaging or manually changing the settings on individual machine if a need to alter firewall settings arises in the future. The GPO firewall settings was set under: Computer Configuration> Policies> Windows Settings> Security Settings> Windows Firewall with Advance Settings. Turn off Screensaver This is handled by the 'Disable_screen_saver” GPO. Set under: Computer Configuration> Policies> Windows Settings> Registry High performance This handled by the deploy steps: “Improve performance reg 1” cmd /c regedit -s "%SCRIPTROOT%\RegEdits\2008HKCUperf.reg" -s Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\PriorityControl] "Win32PrioritySeparation"=dword:00000026 “Improve performance reg 2” cmd /c regedit -s "%SCRIPTROOT%\RegEdits\2008HKCUperf.reg" -s Not going to show the output of this file here due to it is in the neighborhood of 700 lines. Networking After conversation with Releng, there is was no reason to alter network settings. Auto-login Auto login by cltbld is configured by the GPO “Auto_Login” . Set under: Computer Configuration> Policies> Windows Settings> Registry Build tools, compilers and SDKs MozillaBuild 1.5.1 Mozilla Build 1.6.1 was installed. If this need to be changed or upgraded in the future it will be done through GPO. Visual Studio 2010 Do a custom install: C:\Tools\msvs10 Only select "Visual C++" After conversation with Releng It was decided to do a full default install of Visual Studio. The concern was some paths were previous hard coded to a path did not include spaces. For an install with the defualt values the following steps were added to the Build Task Sequence: “Copy Visual Studio 2010 Source Files For specific install” xcopy /s "Z:\Scripts\Microsft Visual Studio 2010\*" "c:\Microsft Visual Studio 2010\" “MSVS 2010 install defualt” "C:\Microsft Visual Studio 2010\Setup\setup.exe" /q /full /norestart “Clean UP MSVS 2010” cmd /c rmdir /q /s "c:\Microsft Visual Studio 2010" Visual Studio 2010 was stalling out when trying to install it as a application through MDT, and it fial when trying to install it from command line ran from a network location. Hence it was copied locally and installed. If the default install does not meet the Releng's needs. It can be installed to specific location with specific packages. For a nonstandard install one will need to create an unattended file. This would be done be logging on to a builder, copy the Visual Studio 2010 files locally and running the following command : "C:\Microsft Visual Studio 2010\Setup\setup.exe" /createunattend c:\unattend.ini This will walk through an installation Wizard which will save the choices made to the unattend.ini (arbitrary name). Then will need to save the file to "Z:\Scripts\Microsft Visual Studio 2010\. Here is the fun part. With in the Visual Studio 2010 files, there is a WCU folder. This folder contains all the prerequisites needed for each package of the install. The full installation will install all of these, but an install using an unattended file does not install these. To further complicate this the exe, msi, and the msu files with in the WCU will not run independently. In order to do an install of specific packages or to specific paths through MDT the following steps would be needed with the assumption the files have been copied locally: Must be first fully installed. MDT run command line step needed: "C:\Microsft Visual Studio 2010\Setup\setup.exe" /q /full /norestart Then it must be uninstalled. MDT run command line step needed: "C:\Microsft Visual Studio 2010\Setup\setup.exe" /q /remove Then reinstalled using the unattended file. MDT run command line step needed: "C:\Microsft Visual Studio 2010\Setup\setup.exe" /full /unattendfile "C:\Microsft Visual Studio 2010\unattend.ini" /norestart Visual Studio 2008 Do a custom installation into C:\Tools\msvs9 Select "Visual C++" (including x64 compilers) Select "Tools for Redistributing Applications" Visual Studio 2008 SP 1 (Installer) Visual Studio 2008 is no longer in use and has been dropped from the MDT build. However if needed the Microsoft Visual Studio 2008 application in MDT will install it with the aforementioned requirements. This installation is done be calling a custom file vs2008pre.bat from the application files: .\WCU\msi31\WindowsInstaller-KB893803-v2-x86.exe /quiet /norestart .\WCU\dotNetFramework\dotnetfx35setup.exe /q /norestart /lang:ENU .\WCU\64bitPrereq\x64\vs_bsln.exe /Q USING_EXUIH=1 .\WCU\DExplore\dexplore.exe /q .\WCU\WebDesignerCore\WebDesignerCore.exe /Q /install sleep 15 .\setup\setup.exe /unattendfile %SCRIPTROOT%\vsunattend.ini In this case the needed components from the WCU were able to be installed independently. DirectX SDK Installed installed as an application in the “Direct X SDK” Build step. Windows 7 SDK The v7.0A SDK was included in the Visual Studio 2010 install. Windows updates Windows updates is handled by the Build step “KB Updates”: cmd /c %SCRIPTROOT%\2008updates.bat wusa %SCRIPTROOT%\Windows6.1-KB2524375-x86.m /q /quiet %SCRIPTROOT%\NDP40-KB2160841-x64.exe /q /norestart %SCRIPTROOT%\NDP40-KB2416472-x64.exe /q /norestart Disable JIT debugger Handled through GPO “Disable_JIT” Computer Configuration> Policies> Windows Settings> Registry Deleting the follow registry values: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\DbgManagedDebugger HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\DbgManagedDebugger Disable Windows Error Reporting Handled by GPO “Disable_windows_reporting” Computer configuration> Policies> Administrative templates> System Internet Communication Management/Internet Communication settings Turn off Windows Error Reporting is enabled. Computer configuration> Policies> Administrative templates> Windows Components/Windows Error Reporting Disable Windows Error Reporting, Display Error Notifications, and Prevent Display of user interface for critical errors are disabled. Add yasm.exe Handled by the Build step “Copy Yasm.exe” cmd /c xcopy "%SCRIPTROOT%/Applications/vsyasm.exe" "c:/mozilla-build/msys/bin" Automation setup Keys This handled and updated by GPO “Files_sshd_testers_builders” Computer Configuration> Windows Settings> Files Copies over the needed files for keys and hosts. Python 2.6 and twisted 9 After conversation with Releng updated Python to a newer version, 2.7.3. The Build application install steps “Python 2.7.3” and “Pywin 218” covers the rest of the items in this catgory. Startup scripts The start-buildbot-win64.bat has been placed in the MDT buildbot files. It also updated by the GPO “Scheduled_tasks_testers_builders”. In the future updating this file will be handled by Puppet. Create directories Create logging and buildbot directories This handled by Run command line Build Step “Create logging directory” mkdir /c/tmp Other tools Installing NRPE Daemon Is not installed in the current version. It will be rolled in the near future through GPO. https://bugzilla.mozilla.org/show_bug.cgi?id=792069 https://bugzilla.mozilla.org/show_bug.cgi?id=798641 SSH Is handled by GPO “Files_sshd_testers_builders” Computer Configuration> Preferences> Windows Settings> Files Copy over the correct files per OS. Computer Configuration> Preferences> Control Panel Settings> Schedule tasks runs immediate task to install KTS. VNC (UltraVNC) This is handled by run command line Build step “Install UltraVNC”: cmd /c regedit -s "%SCRIPTROOT%\RegEdits\2008HKCUperf.reg" -s The VNC configuration is handled by run command line Build step “Replace ultravnc.ini”: cp -f -R "Z:\Scripts\Applications\UltraVNC\ultravnc.ini" "C:\Program Files\uvnc bvba\UltraVnc\" Improve graphics performance This was included with the Deploy step “Improve performance reg 2” Clean up MDT handles disk clean up per install. Buildbot 0.8.4 Copied over the current Buildbot install from the Windows 8 task sequence in the install application Build step “BuildBot- Windows 8” Binscope Handled by the the deploy step “Binscope”. However, this should be moved to the Build task sequence in the future. This was originally in the Build, but the initial install failed without erroring out. Fix auto-login after new password deployment Handled by the aforementioned “Auto_login” GPO. Re-deploy twisted patches No need to re-deployed because of file copy during the MDT process. Deploy Windows SDK for Windows 8 Handled by the install application Build step “Microsoft Win 8 SDK” Deploy the lastest known_host This handled and updated by GPO “Files_sshd_testers_builders” Computer Configuration> Windows Settings> Files Copies over the needed files for keys and hosts. Deploy _dumbwin32proc.py This handled by GPO “File_Mozilla-build Computer Configuration> Policies> Windows Settings> Files Copy files from \\wds1\data-mdt\depshares\ProdShare\Applications\BuildBot_Upgrade\mozilla-build\buildbotve\Lib\site-packages\twisted\internet\_dumbwin*.py to c:\mozilla-build\buildbotve\Lib\site-packages\twisted\internet\_dumbwin32proc.pyc Update to the latest buildbot.bat This handled by the GPO “Scheduled_tasks_testers_builders” Computer Configuration> Policies>Windows Settings> Files Is set to update when the source files change. Deploy the correct keys This handled and updated by GPO “Files_sshd_testers_builders” Computer Configuration> Windows Settings> Files Copies over the needed files for keys and hosts. Deploy google api key This handled by GPO “2008_gapi_key” Computer Configuration> Windows Settings> File Copying from file network location to the local prescribed location. Additionally automatic log on windows have been suppressed via GPO “2008_supress_logon_windows”: Computer Configuration> Policies> Administrative Templates> System/Server Manager> Enabling don not display options. This install is now set up under the Prod Share as “Server 2008 R2 Builder 1”. Added to the Prod Share rules.ini: 2008R2_BUILDER_1] DomainAdmin=ADMINuserName DomainAdminDomain=Domain DomainAdminPassword=ADMINuserNamePassword JoinDomain=releng.ad.mozilla.com MachineObjectOU=OU=builder,OU=SCL3,OU=2008,OU=windows,OU=machines,DC=domain,DC=AD,DC=Mozilla,DC=COM
These machines have allocated for and handed over as prototypes: ix-mn-w0864-001.wintest.releng.scl3.mozilla.com ix-mn-w0864-002.wintest.releng.scl3.mozilla.com
Comment 13 accidentally leaked an old password, so making this bug infra only. Note that the step to create the cltbld account should probably be using the newer password, too.
catlee: looking for you to try out the prototypes and provide feedback/validation that they work as windows builders: ix-mn-w0864-001.wintest.releng.scl3.mozilla.com ix-mn-w0864-002.wintest.releng.scl3.mozilla.com
Whiteboard: [reit] [tracker] [2013Q2] → [reit] [tracker] [2013Q3]
Had a quick look so far and found a couple of problems: 1) We're missing a buildbot.bat file that runs on startup: C:\Users\cltbld\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\buildbot.bat 2) The cltbld home directory is in C:\Users\cltbld.IX-MN-W0864-001 ("echo %userprofile%") while the .ssh directory is under C:\Users\cltbld\.ssh. When I SSH in, I have to disable public key authentication (-o PubkeyAuthentication=no) which is no worse than our existing machines. Then I enter a bash shell prompt: C:> \mozilla-build\msys\bin\bash --login $ echo $HOME /c/Users/cltbld.IX-MN-W0864-001 $ ls ~/.ssh/ ls: /c/Users/cltbld.IX-MN-W0864-001/.ssh/: No such file or directory We're going to want the .ssh directory moved to /Users/cltbld.IX-MN-W0864-001/.ssh or have the cltbld account's home directory be /Users/cltbld instead.
1) This is now called via a startup scheduled tasks directly from the Mozilla-build folder. This has several advantages over launching from the startup folder. Do you have anything hard coded that is calling this path ? 2) We can get this fixed in pretty short order. I will sync up with Mark today
No, nothing is relying on that specific path to my knowledge. The default bash PATH differs on the new machine versus existing: cltbld@W64-IX-SLAVE20 ~ $ echo $PATH .:/usr/local/bin:/mingw/bin:/bin:/c/windows/system32:/c/windows:/c/windows/System32/Wbem:/c/windows/System32/Wi ndowsPowerShell/v1.0/:/c/mozilla-build/python:/usr/bin:/c/mozilla-build/vim/vim72:/c/Program Files (x86)/Window s Kits/8.0/Windows Performance Toolkit/:. cltbld@IX-MN-W0864-001 ~ $ echo $PATH .:/usr/local/bin:/mingw/bin:/bin:/c/windows/system32:/c/windows:/c/windows/System32/Wbem:/c/windows/System32/Wi ndowsPowerShell/v1.0/:/c/mozilla-build/python27:/c/mozilla-build/python27/Scripts:/usr/bin:/c/mozilla-build/vim /vim72:/c/mozilla-build/wget:/c/mozilla-build/info-zip:/c/CoreUtils/bin:/c/mozilla-build/buildbotve/scripts:/c/ Program Files (x86)/Microsoft SQL Server/100/Tools/Binn/:/c/Program Files/Microsoft SQL Server/100/Tools/Binn/: /c/Program Files/Microsoft SQL Server/100/DTS/Binn/:/c/Program Files (x86)/Windows Kits/8.0/Windows Performance Toolkit/:.
Successfully built mozilla-central Firefox from the command-line. The only tweak I had to make to the PATH set by start-msvc2010.bat was to add /c/mozilla-build/python27 Next, I'll try and hook one of these build slaves up to a staging master.
We can append that to the system path via GPO
I see two scheduled tasks that both point to c:\mozilla-build\start-buildbot-win64.bat. They appear to have the same startup conditions - I think we only need one. Also, the c:\mozilla-build\start-buildbot-win64.bat file does not exist.
There should only be one scheduled task now. The taloslave schedule task should no longer be applied by GPO to this OS. Looking into what is going on with the start-buildbot-win64.bat file now.
The start-buildbot-win64.bat file is now showing up the correct location through GPO.
The startup script above is failing with "Microsoft Visual C++ version 9 (2008) was not found. Exiting." I compared against the original buildbot.bat (called from the Start menu's startup items) and besides the failure I just described, there are some things that the new script isn't doing like cleaning out temporary files and sleeping 20 seconds on startup to let things settle.
So we have two separate issues to address here. The first is we have a start script that then calls a start script. For now we will use that same methodology but we should consider wrapping everything up at some point. The second is that the first wrapper script buildbot.bat ( called at startup ) calls start-buildbot.bat the current version in puppet (and on the current 2008 machines) checks for 2008 with this code: if "%VC9DIR%"=="" ( if "%VC9EXPRESSDIR%"=="" ( ECHO "Microsoft Visual C++ version 9 (2008) was not found. Exiting." pause EXIT /B 1 ) In a meeting at the beginning of this project we were told that this bat files would be edited so that 2008 would not be checked for anymore. Who would be responsible for redoing the batch file ?
Status: REOPENED → ASSIGNED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
1) I'm going to take care of making the necessary batch file changes. 2) Can we use the old install path for MSVS10? New path: c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ Old path: C:\Tools\msvs10\Common7\IDE\ The new path conflicts with our setting in http://hg.mozilla.org/mozilla-central/file/5f79304fcefe/build/win64/mozconfig.vs2010 I think we could add both paths to the mozconfig during the transition but it would be cleaner to have all win64 build slaves use the same path. 3) Can you please remove the Microsoft F# installation in: c:\Program Files (x86)\Microsoft F#\v4.0
jhopkins: Regarding point 2, we had a meeting previously (which you probably weren't caught up on and are coming into this without background) to talk about MSVS. The decision between releng/ateam/relops was to install it in the default MS location and then make modifications to the current code to try to make that work. MS is usually pretty pick about where it installs things, so this saves a lot of future headache for any updates or future installs.
For number 3 are we talking about removing the entire directory or just the "Microsoft F#" folder?
arr: ok no problem, we'll find a way to deal with the differing path. You're right - I wasn't at that meeting and am lacking some context. markco: I'd prefer if at install/imaging time we simply did not install Microsoft F#. That way nothing in the registry points at that directory, there's no F# DLL's installed to the system directory, etc.. 4) We were missing c:\runslave.py which I copied manually. Please add that file to the configuration.
I will look into 3 and 4 this afternoon.
I think I got F# removed from the install without affecting the rest of the MSVS 10 installation. I kicking off an install to do a capture of the build now. Tomorrow morning I will test the deploy task sequence with the new captured image.
markco: how difficult would it be to update the VC++2010 runtime to MS11-025, per bug 891659?
Looks like that is done through an update (KB2565063), so it should not be too difficult. If it is not already in the install, I will take a look at this after noon.
6) Great - let's call the "upgrade VC++2010 runtime to MS11-025" item #6 7) Can you please have the contents of this zip file extracted into C:\ (note: I've already done it manually on ix-mn-w0864-001)  http://people.mozilla.org/~jhopkins/bug781277/psutil-0.7.1.win32.zip
Created attachment 779191 [details] start-buildbot-win64.bat 8) Replace the existing c:\mozilla-build\start-buildbot-win64.bat with this file and point the startup task at it.
1) Handled previously. 2) Discussed previously. 3) Microsoft F# has been removed from the build. 1. The F# installation files was within the WCU (prerequisites and optional package folder) folder of the extracted MSVS installation package. Removed the F# files and allowed the MSVS 10 install MDT step to continue on error. 4) Will be handled by a pathing GPO 5) Handled by the file local users GPO 6) VC++2010 runtime to MS11-025 1. This was part of the KB2565063 update (vcredist_x64.exe ). This updated was added to the 2008updates.bat that is called in the KB updates in the build task sequence. 1. %SCRIPTROOT%\vcredist_x64.exe /q 7) The psutil folder in the above zip file has been added to python files that are copied over during the build step. 8) Buildbot.bat has been replaced with the above file, https://bug781277.bugzilla.mozilla.org/attachment.cgi?id=779191 1. This file is already set up in a schedule task through GPO.
Added MAPI header step to the build task sequence: cmd /c xcopy "z:\files\MAPI headers\*.*" "C:\Office 2010 Developer Resources\Outlook 2010 MAPI Headers\" (https://bugzilla.mozilla.org/show_bug.cgi?id=891956) Recapturing the image and then will test with the deploy task sequence.
jhopkins: I just kicked off installs on the test machines. They should now include more complete Python27 directories. The two files you mentioned twistd.bat and twistd.py were in a different Python directory, so I copied those files into the Python27 directory. Let me know if that worked, please.
1) Take new c:\mozilla-build\runslave.py into the image/GPO 2) $HOME/.hgrc is misnamed ".hrgc" 3) Missing files under C:\mozilla-build\buildbotve, including scripts\activate.bat On an existing w64 build slave, running "activate.bat", then "pip freeze" says: Twisted==10.2.0 buildbot==0.8.4-pre-moz2 buildbot-slave==0.8.4-pre-moz2 distribute==0.6.14 pywin32==214 simplejson==2.1.3 zope.interface==3.6.1
1) The file has been updated and the change will be picked up by GPO. 2)Spelling error has been corrected in the source files for the GPO. Will be corrected by GPO> 3)Diving deeper to find out what happened with those files during the install process.
3)During the buildbot install task sequence step either directories are being over-written or just deleted. I will need to rework the buildbot.bat for 2008.I'll up date the ticket once I have a new install.
The C:\mozilla-build\buildbotve\Scripts is now there. C:\mozilla-build\buildbotve\scripts>C:\mozilla-build\buildbotve\scripts\activate .bat (buildbotve) C:\mozilla-build\buildbotve\scripts>pip.exe freeze Twisted==10.2.0 buildbot==0.8.4-pre-moz2 buildbot-slave==0.8.4-pre-moz2 distribute==0.6.14 pywin32==214 simplejson==2.1.3 The 2 test machines have been re-imaged and are ready for testing. The reason it is taking so long between these iterations is because these files are captured within the wim. In the future these files will be handled Puppet eliminated the need to capture them within the wim. zope.interface==3.6.1
Thanks, Mark! Things are looking much better now. You may need to verify the VNC password is correct - I had to use an older password earlier today but the latest password works now. I think we're done the bulk of the configuration work and can start to focus our efforts on getting this into AWS. There may be a configuration tweak or two along the way but no need for us to block on getting it perfect.
Running into intermittent conftest.exe "permission denied" failures while doing a WINNT 5.2 mozilla-inbound pgo-build: ====== checking whether the C++ compiler (cl ) works... no ------ config.log ------ rm: cannot lstat `conftest.exe': Permission denied configure: error: installation or configuration problem: C++ compiler cannot create executables. ====== A possible solution (requiring GPO changes) is described at https://bugzilla.mozilla.org/show_bug.cgi?id=545015#c24 Should do more investigation as to what differs between production w64-ix and the new ix-mn build slaves that would cause this intermittent problem.
Link to mozilla.dev.platform post initial cutover plan: https://groups.google.com/forum/#!topic/mozilla.dev.platform/zACrUe_JwKw We hit a blocker this week in bug 925894 which has been resolved. I will be sending out an updated win64-rev2 migration proposal today to Sheriffs, QA, A-Team, RelEng, and RelOps.
Currently working through staging mozilla-beta l10n repack timeouts in bug 928982. As proposed on the forum, we will start with two rev2 build machines mixed into the rev1 try pool and let that build for a ~week, then mozilla-central+inbound for a week, then look at aurora/beta/release/esr. I don't expect larger scale machine reimaging to happen for another 2 weeks or so.
Waiting until Monday to deploy two win64-rev2 build slaves onto try and try-comm-central so as not to bitrot an unrelated, complex uplift patch scheduled to land on Monday (per rail).
Patch for mixed win64 rev1/rev2 pool on 'try' (bug 930506) landed and reconfigured.
Mixed win64 rev1/rev2 try pool is enabled and the first rev2-based try build is running.
No issues found with the Try builds. :philor was happy with the test results as well. mozilla-inbound and b2g-inbound branches will be switched to a mixed rev1/rev2 pool later today.
Next week, we plan to migrate 40 each of win64-rev1 try build and production build slaves to win64-rev2. Timetable: Monday: - jhopkins will buildbot-configs patch, split hosts in etherpad into try and production batches, and check builds run over the weekend - markco will verify hosts that need a reboot for ipmi - jhopkins will disable hosts in off-peak hours in advance of the reimage to ensure they are idle by the time of reimaging Tuesday to Friday: - 7:00am Pacific: markco will ping jhopkins before he starts reimaging, then kick off reimages for 5 of each type - 12:00pm Pacific: same procedure as 7:00am, with 5 different slaves of each type Note: RelEng will not be blocked on relops if builds do not finish in time since they can also reimage machines as needed.  https://etherpad.mozilla.org/win64-rev2
This week's cutovers are being tracked in bug 934976. So far, things are going very well.
The migration of 80 rev1 w64-ix-slave to rev2 (bug 934976) is complete. I'll follow up with next steps on Monday.
Planning to switch mozilla-central and comm-central to a mixed win64-rev1 + win64-rev2 pool tomorrow, per https://groups.google.com/forum/#!msg/mozilla.dev.platform/VSgwbVIuDwg/2a3y3gGq0OIJ
try/try-comm-central are now building only on win64-rev2. Builds are looking good. mozilla-central and comm-central are configured to use a mixed pool of win64-rev1 and win64-rev2 build machines. m-c is currently closed for other reasons, so it will take longer to verify that things are working as expected.
Updated status: * mozilla-central and comm-central branches are building only on win64-rev2 * mozilla-inbound and b2g-inbound remain on a mixed rev1/rev2 pool to ensure an adequate number of build slaves to meet its high checkin load * 24 win64-rev1 build machines are left to reimage, pending cutover of aurora, beta, release, esr branches. * Firefox desktop nightlies are building on win64-rev2 since Tuesday, November 27 What's next: * start building aurora on win64-rev2 in 1 week * start building beta on win64-rev2 for beta 1
Now that we've verified that the new imaging/management system is working, this goal is complete. Please open separate bugs to handle additional machine reimagings.
Note: Bug 918414 will be used to track the remaining Release Engineering work.
You need to log in before you can comment on or make changes to this bug.