781277 - created MDT-driven automated install configuration for windows 2008R2 (w64) builders

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Matthew Larrain[:MaRu]]

This is a tracking bug for the W64 project for automating the installing of applications and systems changes.

Comment 1

[Matthew Larrain[:MaRu]]

Attempting to build an MSI package of VS2010 to see if that fixes the install issue we are running into with this.

Comment 2

[Matthew Larrain[:MaRu]]

Was able to deploy Visual Studio 2010 via MDT. 
Moving on to finishing the last updates needed for this image and we should be able to start deploying with it shortly.

Comment 3

[Matthew Larrain[:MaRu]]

Last night I was able to push the hard drive improvements per the wiki using a command like script of; 

fsutil behavior set disable8dot3 1 && fsutil behavior set disablelastaccess 1

Able to change the pagefile size via the command line and am tweaking it to run via automation using; 

wmic.exe computersystem set AutomaticManagedPagefile=False && wmic.exe pagefileset set InitialSize=4048,MaximumSize=4048

Setting the power options via command line requires getting the GUID from the system using powercfg after that the cli script looks like; 

powercfg -setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c

Comment 4

[Matthew Larrain[:MaRu]]

mozilla tools installs correctly now

Comment 5

[Matthew Larrain[:MaRu]]

Got the pagefile to actually work ran into an issue where it didn't want to run from MDT itself so I wrote a batch script that combines pagefile and powercfg into one. Working on getting the ssh keys into the right place. Per conversation with dustin we are just moving them from a folder on the MDT box and copying them across to the .ssh folder.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I'd like to hear a definitive answer on whether we should be installing production ssh keys with MDT.  The share on MDT is password authenticated, so it's trustworthy.

Comment 7

[u429623]

(correct dependency - not win8 related)

We'll get an answer on the keys. The current keys installed are old, and not the latest production ones. Installing the fresh keys is the major step RelEng does prior to enabling the device

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I got an answer from coop, actually - we should install the staging keys, and releng can do the prod keys manually.

Comment 9

[Matthew Larrain[:MaRu]]

Writing up process list for discussion on what to use MDT for vs what to use GPO for. Meeting scheduled with Dustin for tomorrow to review and update documentation. This is for both builders and testers.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 827497 is the last stretch of getting the MDT task sequence that starts with a capture to actually work.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Looks good

Comment 12

[Amy Rich [:arr] [:arich]]

This bug is actually for Throop, not Halsted and should therefore remain open as a tracker until Throop is complete.

Comment 13

[Mark Cornmesser [:markco]]

Started with the Server 2008 R2 clean which had been set up by our consultant. 

Replaced the OS with a version of 2008 R2 that included Service Pack 1. 

The Task sequence was set up according the info located at: 

https://wiki.mozilla.org/ReferencePlatforms/Win64


The following are the individual items from that page and how they were addressed: 

Administrator password
The administrator password is original set during the creation of the task sequence in MDT. It is then later controlled by GPO , User_Local_Builder. This GPO was created by copying the User_local_tester  users data and adding cltbld user to the local administrator group. That was done to do to various items that is needed to be done by this user that is locked down to the administrator group on Server 2008. 

Virtual Memory 

Un-check "Automatically manage paging file size for all drives" 
Custom size: 4048-4048 

This was handled by the “Paging File Size” step in the build task Sequence. Reg file located at: 
%SCRIPTROOT%\RegEdits\2008memPage.reg" 

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
"ClearPageFileAtShutdown"=dword:00000000
"DisablePagingExecutive"=dword:00000000
"LargeSystemCache"=dword:00000001
"NonPagedPoolQuota"=dword:00000000
"NonPagedPoolSize"=dword:00000000
"PagedPoolQuota"=dword:00000000
"PagedPoolSize"=dword:00000000
"SecondLevelDataCache"=dword:00000000
"SessionPoolSize"=dword:00000004
"SessionViewSize"=dword:00000030
"SystemPages"=dword:00000000
"PagingFiles"=hex(7):63,00,3a,00,5c,00,70,00,61,00,67,00,65,00,66,00,69,00,6c,\
  00,65,00,2e,00,73,00,79,00,73,00,20,00,34,00,30,00,34,00,38,00,20,00,34,00,\
  30,00,34,00,38,00,00,00,00,00
"PhysicalAddressExtension"=dword:00000001
"ExistingPageFiles"=hex(7):5c,00,3f,00,3f,00,5c,00,43,00,3a,00,5c,00,70,00,61,\
  00,67,00,65,00,66,00,69,00,6c,00,65,00,2e,00,73,00,79,00,73,00,00,00,00,00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters]
"BootId"=dword:00000008
"BaseTime"=dword:16c177f2

Hard-drive adjustments

Based off of bug :https://bugzilla.mozilla.org/show_bug.cgi?id=798639 and a conversation with Releng it was decided, due to the available disk space on the current hardware, that the builders will only configured with a single drive, the C: drive. 

Clean Manager

Clean Manger is installed using the servermanagercmd in the “Enable Clean Manger” build step:
cmd /c servermanagercmd -install Desktop-Experience -resultPath results.xml


Daemon tools
Based in a conversation with Releng, Daemon Tools was only needed to manually set up a builder. It was used to mount the Visual Studio ISOs. However, if Daemon Tools is need in the future. It is now an available application in MDT. 


Create cltbld user 
User creation was copied over from previous task sequences, “Create cltbld account” Build step. It runs the following command :
 net user cltbld xxxPASSxxx /add


Basic OS Setup 
Verify the name of the NetBios computer name and the DNS suffix are correct
This handled by the current naming script that is used with all the MDT installs. 

 Check for updates 
Check for updates a second time after the first batch of updates are installed 
Windows Server 2008 R2 Service Pack 1 x64 Edition (KB976932) 
This was handled by the OS version that was imported and used for the task sequences.
 
Disable automatic updates 
This is handled by the “Disable_auto_updates” GPO. Which under:
Computer Configuration> Policies> Administrative Templates > Windows Components /Windows Update, configure automatic updates is disabled.  

Keep Windows Firewall ON and add exceptions once you install the SSH and VNC servers
This handled by the build steps using the netsh cmd: 
“Allow sshtunnel through firewall”
netsh advfirewall firewall add rule name="Allow SSH tunnel" dir=in action=allow program="C:\slave\test\build\bin\ssltunnel.exe"

“Allow VNC through firewall”
netsh advfirewall firewall add rule name="Allow VNC" dir=in action=allow program="C:\Program Files\uvnc bvba\ultravnc\winvnc.exe"

A firewall acceptation was set for Python as well: 
“Allow python through firewall”
netsh advfirewall firewall add rule name="Allow Python" dir=in action=allow program="C:\mozilla-build\buildbotve\scripts\python.exe"
The firewall is also managed through GPO, “Firewall_exceptions_testers_builders”. This was done so that firewalls setting may be managed without re-imaging or manually changing the settings on individual machine if a need to alter firewall settings arises in the future. The GPO firewall settings was set under:
Computer Configuration> Policies> Windows Settings> Security Settings> Windows Firewall with Advance Settings. 

Turn off Screensaver
This is handled by the 'Disable_screen_saver” GPO. Set under:
Computer Configuration> Policies> Windows Settings> Registry 

High performance
This handled by the deploy steps: 
“Improve performance reg 1” 
cmd /c regedit -s "%SCRIPTROOT%\RegEdits\2008HKCUperf.reg" -s
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\PriorityControl]
"Win32PrioritySeparation"=dword:00000026

“Improve performance reg  2”
cmd /c regedit -s "%SCRIPTROOT%\RegEdits\2008HKCUperf.reg" -s
Not going to show the output of this file here due to it is in the neighborhood of 700 lines.

Networking
After conversation with Releng, there is was no reason to alter network settings. 

Auto-login 
Auto login by cltbld is configured by the GPO “Auto_Login” . Set under:
Computer Configuration> Policies> Windows Settings> Registry

Build tools, compilers and SDKs 
MozillaBuild 1.5.1 
Mozilla Build 1.6.1 was installed. If this need to be changed or upgraded in the future it will be done through GPO. 

Visual Studio 2010 

Do a custom install: 
C:\Tools\msvs10 
Only select "Visual C++" 
After conversation with Releng It was decided to do a  full default install of Visual Studio. The concern was some paths were previous hard coded to a path did not include spaces. 
For an install with the defualt values the following steps were added to the Build Task Sequence:
“Copy Visual Studio 2010  Source Files  For specific install”
xcopy /s "Z:\Scripts\Microsft Visual Studio 2010\*" "c:\Microsft Visual Studio 2010\"

“MSVS 2010 install  defualt”
"C:\Microsft Visual Studio 2010\Setup\setup.exe"  /q /full /norestart

“Clean UP MSVS 2010”
cmd /c rmdir /q /s  "c:\Microsft Visual Studio 2010"

Visual Studio 2010 was stalling out when trying to install it as a application through MDT, and it fial when trying to install it from command line ran from a network location. Hence it was copied locally and installed. 

If the default install does not meet the Releng's needs. It can be installed to specific location with specific packages. For a nonstandard install one will need to create an unattended file. This would be done be logging on to a builder, copy the Visual Studio 2010 files locally and running the following command : 
"C:\Microsft Visual Studio 2010\Setup\setup.exe"  /createunattend c:\unattend.ini
This will walk through an installation Wizard which will save the choices made to the unattend.ini (arbitrary name). Then will need to save the file to  "Z:\Scripts\Microsft Visual Studio 2010\.

Here is the fun part. With in the Visual Studio 2010 files, there is a WCU folder. This folder contains all the prerequisites needed for each package of the install. The full installation will install all of these, but an install using an unattended file  does not install these. To further complicate this the exe, msi, and the msu files with in the WCU will not run independently. In order to do an install of specific packages or to specific paths through MDT the following steps would be needed with the assumption the files have been copied locally: 
Must be first fully installed. MDT run command line step needed:  
"C:\Microsft Visual Studio 2010\Setup\setup.exe"  /q /full /norestart

Then it must be uninstalled. MDT run command line step needed:
"C:\Microsft Visual Studio 2010\Setup\setup.exe"  /q /remove

Then reinstalled using the unattended file.  MDT run command line step needed:
"C:\Microsft Visual Studio 2010\Setup\setup.exe"  /full /unattendfile "C:\Microsft Visual Studio 2010\unattend.ini" /norestart

Visual Studio 2008 
Do a custom installation into C:\Tools\msvs9 
Select "Visual C++" (including x64 compilers) 
Select "Tools for Redistributing Applications" 
Visual Studio 2008 SP 1 (Installer)
Visual Studio 2008 is no longer in use and has been dropped from the MDT build. However if needed the Microsoft Visual Studio 2008 application in MDT will install it with the aforementioned requirements. This installation is done be calling a custom file vs2008pre.bat from the application files:
.\WCU\msi31\WindowsInstaller-KB893803-v2-x86.exe /quiet /norestart

.\WCU\dotNetFramework\dotnetfx35setup.exe /q /norestart /lang:ENU

.\WCU\64bitPrereq\x64\vs_bsln.exe /Q USING_EXUIH=1

.\WCU\DExplore\dexplore.exe /q
.\WCU\WebDesignerCore\WebDesignerCore.exe /Q /install

sleep 15

.\setup\setup.exe /unattendfile  %SCRIPTROOT%\vsunattend.ini

In this case the needed components from the WCU were able to be installed independently.

DirectX SDK
Installed installed as an application in the “Direct X SDK” Build step. 

Windows 7 SDK 
The v7.0A SDK was included in the Visual Studio 2010 install. 

Windows updates 
Windows updates is handled by the Build step “KB Updates”:
cmd /c %SCRIPTROOT%\2008updates.bat

wusa %SCRIPTROOT%\Windows6.1-KB2524375-x86.m /q /quiet

%SCRIPTROOT%\NDP40-KB2160841-x64.exe /q /norestart
 
%SCRIPTROOT%\NDP40-KB2416472-x64.exe /q /norestart

Disable JIT debugger 
Handled through GPO “Disable_JIT”
Computer Configuration> Policies> Windows Settings> Registry
Deleting the follow registry values:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\DbgManagedDebugger
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\DbgManagedDebugger

Disable Windows Error Reporting 
Handled by GPO “Disable_windows_reporting”
Computer configuration> Policies> Administrative templates> System Internet Communication Management/Internet Communication settings
Turn off Windows Error Reporting is enabled.
Computer configuration> Policies> Administrative templates> Windows Components/Windows Error Reporting
Disable Windows Error Reporting, Display Error Notifications,  and Prevent Display of user interface for critical errors are disabled. 
Add yasm.exe 
Handled by the Build step “Copy Yasm.exe”
cmd /c xcopy  "%SCRIPTROOT%/Applications/vsyasm.exe" "c:/mozilla-build/msys/bin"
Automation setup 
Keys 
This handled and updated by GPO “Files_sshd_testers_builders”
Computer Configuration> Windows Settings> Files Copies over the needed files for keys and hosts. 
Python 2.6 and twisted 9 

After conversation with Releng updated Python to a newer version, 2.7.3. 
The Build application install steps “Python 2.7.3” and “Pywin 218” covers the rest of the items in this catgory. 


Startup scripts
The start-buildbot-win64.bat has been placed in the MDT buildbot files. It also updated by the GPO “Scheduled_tasks_testers_builders”.  In the future updating this file will be handled by Puppet. 


Create directories
Create logging and buildbot directories
This handled by Run command line Build Step “Create logging directory”
mkdir /c/tmp

Other tools 
Installing NRPE Daemon 
Is not installed in the current version. It will be rolled in the near future through GPO. 
https://bugzilla.mozilla.org/show_bug.cgi?id=792069
https://bugzilla.mozilla.org/show_bug.cgi?id=798641


SSH 
Is handled by GPO “Files_sshd_testers_builders”
Computer Configuration> Preferences> Windows Settings> Files Copy over the correct files per OS.
Computer Configuration> Preferences> Control Panel Settings> Schedule tasks runs immediate task to install KTS.   
VNC (UltraVNC) 
This is handled by run command line Build step “Install UltraVNC”:
cmd /c regedit -s "%SCRIPTROOT%\RegEdits\2008HKCUperf.reg" -s

The VNC configuration is handled by run command line Build step “Replace ultravnc.ini”:
 cp -f -R "Z:\Scripts\Applications\UltraVNC\ultravnc.ini" "C:\Program Files\uvnc bvba\UltraVnc\"   

Improve graphics performance 
This was included with the Deploy step “Improve performance reg  2”

Clean up 
MDT handles disk clean up per install. 

Buildbot 0.8.4 
Copied over the current Buildbot install from the Windows 8 task sequence in the install application Build step “BuildBot- Windows 8”
Binscope 
Handled by the the deploy step “Binscope”. However, this should be moved to the Build task sequence in the future. This was originally in the Build, but the initial install failed without erroring out.  


Fix auto-login after new password deployment 
Handled by the aforementioned “Auto_login” GPO.

Re-deploy twisted patches 
No need to re-deployed because of file copy during the MDT process. 


Deploy Windows SDK for Windows 8 

Handled by the install application Build step “Microsoft Win 8 SDK”

Deploy the lastest known_host
This handled and updated by GPO “Files_sshd_testers_builders”
Computer Configuration> Windows Settings> Files Copies over the needed files for keys and hosts. 




 
Deploy _dumbwin32proc.py 
This handled by GPO “File_Mozilla-build
Computer Configuration> Policies> Windows Settings> Files Copy files from \\wds1\data-mdt\depshares\ProdShare\Applications\BuildBot_Upgrade\mozilla-build\buildbotve\Lib\site-packages\twisted\internet\_dumbwin*.py to c:\mozilla-build\buildbotve\Lib\site-packages\twisted\internet\_dumbwin32proc.pyc


Update to the latest buildbot.bat 
This handled by the GPO “Scheduled_tasks_testers_builders”
Computer Configuration> Policies>Windows Settings> Files Is set to update when the source files change.  

Deploy the correct keys 
This handled and updated by GPO “Files_sshd_testers_builders”
Computer Configuration> Windows Settings> Files Copies over the needed files for keys and hosts. 

Deploy google api key 
This handled by GPO “2008_gapi_key”
Computer Configuration> Windows Settings> File 
Copying from file network location to the local prescribed location. 

Additionally automatic log on windows have been suppressed via GPO “2008_supress_logon_windows”:

Computer Configuration> Policies> Administrative Templates> System/Server Manager> 
Enabling don not display options. 

This install is now set up under the Prod Share as “Server 2008 R2 Builder 1”. 
Added to the Prod Share rules.ini:
2008R2_BUILDER_1]
DomainAdmin=ADMINuserName
DomainAdminDomain=Domain
DomainAdminPassword=ADMINuserNamePassword
JoinDomain=releng.ad.mozilla.com
MachineObjectOU=OU=builder,OU=SCL3,OU=2008,OU=windows,OU=machines,DC=domain,DC=AD,DC=Mozilla,DC=COM

Comment 14

[Mark Cornmesser [:markco]]

These machines have allocated for and handed over as prototypes: 
ix-mn-w0864-001.wintest.releng.scl3.mozilla.com
ix-mn-w0864-002.wintest.releng.scl3.mozilla.com

Comment 15

[Amy Rich [:arr] [:arich]]

Comment 13 accidentally leaked an old password, so making this bug infra only.  Note that the step to create the cltbld account should probably be using the newer password, too.

Comment 16

[Amy Rich [:arr] [:arich]]

catlee: looking for you to try out the prototypes and provide feedback/validation that they work as windows builders:

ix-mn-w0864-001.wintest.releng.scl3.mozilla.com
ix-mn-w0864-002.wintest.releng.scl3.mozilla.com

Comment 17

[John Hopkins (:jhopkins)]

Had a quick look so far and found a couple of problems:

1) We're missing a buildbot.bat file that runs on startup:

C:\Users\cltbld\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\buildbot.bat

2) The cltbld home directory is in C:\Users\cltbld.IX-MN-W0864-001 ("echo %userprofile%") while the .ssh directory is under C:\Users\cltbld\.ssh.

When I SSH in, I have to disable public key authentication (-o PubkeyAuthentication=no) which is no worse than our existing machines.  Then I enter a bash shell prompt:
C:> \mozilla-build\msys\bin\bash --login
$ echo $HOME
/c/Users/cltbld.IX-MN-W0864-001
$ ls ~/.ssh/
ls: /c/Users/cltbld.IX-MN-W0864-001/.ssh/: No such file or directory

We're going to want the .ssh directory moved to /Users/cltbld.IX-MN-W0864-001/.ssh or have the cltbld account's home directory be /Users/cltbld instead.

Comment 18

[Q]

1) This is now called via a startup scheduled tasks directly from the Mozilla-build folder. This has several advantages over launching from the startup folder. Do you have anything hard coded that is calling this path ? 

2) We can get this fixed in pretty short order. I will sync up with Mark today

Comment 19

[John Hopkins (:jhopkins)]

No, nothing is relying on that specific path to my knowledge.


The default bash PATH differs on the new machine versus existing:

cltbld@W64-IX-SLAVE20 ~
$ echo $PATH
.:/usr/local/bin:/mingw/bin:/bin:/c/windows/system32:/c/windows:/c/windows/System32/Wbem:/c/windows/System32/Wi
ndowsPowerShell/v1.0/:/c/mozilla-build/python:/usr/bin:/c/mozilla-build/vim/vim72:/c/Program Files (x86)/Window
s Kits/8.0/Windows Performance Toolkit/:.

cltbld@IX-MN-W0864-001 ~
$ echo $PATH
.:/usr/local/bin:/mingw/bin:/bin:/c/windows/system32:/c/windows:/c/windows/System32/Wbem:/c/windows/System32/Wi
ndowsPowerShell/v1.0/:/c/mozilla-build/python27:/c/mozilla-build/python27/Scripts:/usr/bin:/c/mozilla-build/vim
/vim72:/c/mozilla-build/wget:/c/mozilla-build/info-zip:/c/CoreUtils/bin:/c/mozilla-build/buildbotve/scripts:/c/
Program Files (x86)/Microsoft SQL Server/100/Tools/Binn/:/c/Program Files/Microsoft SQL Server/100/Tools/Binn/:
/c/Program Files/Microsoft SQL Server/100/DTS/Binn/:/c/Program Files (x86)/Windows Kits/8.0/Windows Performance
 Toolkit/:.

Comment 20

[John Hopkins (:jhopkins)]

Successfully built mozilla-central Firefox from the command-line.  The only tweak I had to make to the PATH set by start-msvc2010.bat was to add /c/mozilla-build/python27

Next, I'll try and hook one of these build slaves up to a staging master.

Comment 21

[Q]

We can append that to the system path via GPO

Comment 22

[John Hopkins (:jhopkins)]

I see two scheduled tasks that both point to c:\mozilla-build\start-buildbot-win64.bat.  They appear to have the same startup conditions - I think we only need one.

Also, the c:\mozilla-build\start-buildbot-win64.bat file does not exist.

Comment 23

[Mark Cornmesser [:markco]]

There should only be one scheduled task now. The taloslave schedule task should no longer be applied by GPO to this OS. 

Looking into what is going on with the start-buildbot-win64.bat file now.

Comment 24

[Mark Cornmesser [:markco]]

The start-buildbot-win64.bat file is now showing up the correct location through GPO.

Comment 25

[John Hopkins (:jhopkins)]

The startup script above is failing with "Microsoft Visual C++ version 9 (2008) was not found. Exiting."
I compared against the original buildbot.bat (called from the Start menu's startup items) and besides the failure I just described, there are some things that the new script isn't doing like cleaning out temporary files and sleeping 20 seconds on startup to let things settle.

Comment 26

[Q]

So we have two separate issues to address here. The first is we have a start script that then calls a start script. For now we will use that same methodology but we should consider wrapping everything up at some point. 

 The second is that the first wrapper script buildbot.bat ( called at startup ) calls start-buildbot.bat the current version in puppet (and on the current 2008 machines) checks for 2008 with this code:

if "%VC9DIR%"=="" (
    if "%VC9EXPRESSDIR%"=="" (
        ECHO "Microsoft Visual C++ version 9 (2008) was not found. Exiting."
        pause
        EXIT /B 1
    )

In a meeting at the beginning of this project we were told that this bat files would be edited so that 2008 would not be checked for anymore. Who would be responsible for redoing the batch file ?

Comment 27

[John Hopkins (:jhopkins)]

1) I'm going to take care of making the necessary batch file changes.

2) Can we use the old install path for MSVS10?

New path:
 c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\
Old path:
 C:\Tools\msvs10\Common7\IDE\

The new path conflicts with our setting in http://hg.mozilla.org/mozilla-central/file/5f79304fcefe/build/win64/mozconfig.vs2010

I think we could add both paths to the mozconfig during the transition but it would be cleaner to have all win64 build slaves use the same path.

3) Can you please remove the Microsoft F# installation in:

 c:\Program Files (x86)\Microsoft F#\v4.0

Comment 28

[Amy Rich [:arr] [:arich]]

jhopkins: Regarding point 2, we had a meeting previously (which you probably weren't caught up on and are coming into this without background) to talk about MSVS.  The decision between releng/ateam/relops was to install it in the default MS location and then make modifications to the current code to try to make that work.  MS is usually pretty pick about where it installs things, so this saves a lot of future headache for any updates or future installs.

Comment 29

[Mark Cornmesser [:markco]]

For number 3 are we talking about removing the entire directory or just the "Microsoft F#" folder?

Comment 30

[John Hopkins (:jhopkins)]

arr: ok no problem, we'll find a way to deal with the differing path.  You're right - I wasn't at that meeting and am lacking some context.

markco: I'd prefer if at install/imaging time we simply did not install Microsoft F#.  That way nothing in the registry points at that directory, there's no F# DLL's installed to the system directory, etc..

4) We were missing c:\runslave.py which I copied manually.  Please add that file to the configuration.

Comment 31

[Mark Cornmesser [:markco]]

I will look into 3 and 4 this afternoon.

Comment 32

[Mark Cornmesser [:markco]]

I think I got F# removed from the install without affecting the rest of the MSVS 10 installation. 

I kicking off an install to do a capture of the build now. Tomorrow morning I will test the deploy task sequence with the new captured image.

Comment 33

[John Hopkins (:jhopkins)]

Attachment: $HOME/.hgrc

5) We'll need this file deployed.

Comment 34

[John Hopkins (:jhopkins)]

markco: how difficult would it be to update the VC++2010 runtime to MS11-025, per bug 891659?

Comment 35

[Mark Cornmesser [:markco]]

Looks like that is done through an update (KB2565063), so it should not be too difficult. If it is not already in the install, I will take a look at this after noon.

Comment 36

[John Hopkins (:jhopkins)]

6) Great - let's call the "upgrade VC++2010 runtime to MS11-025" item #6

7) Can you please have the contents of this zip file[1] extracted into C:\ (note: I've already done it manually on ix-mn-w0864-001)

[1] http://people.mozilla.org/~jhopkins/bug781277/psutil-0.7.1.win32.zip

Comment 37

[John Hopkins (:jhopkins)]

Attachment: start-buildbot-win64.bat

8) Replace the existing c:\mozilla-build\start-buildbot-win64.bat with this file and point the startup task at it.

Comment 38

[Mark Cornmesser [:markco]]

1) Handled previously.
2) Discussed previously.
3) Microsoft F# has been removed from the build.
1. The F# installation files was within the WCU (prerequisites and optional package folder) folder of the extracted MSVS  installation package. Removed the F# files and allowed the MSVS 10 install MDT step to continue on error.
4) Will be handled by a pathing GPO
5) Handled by the file local users GPO
6) VC++2010 runtime to MS11-025
1. This was part of the KB2565063 update (vcredist_x64.exe ). This updated was added to the 2008updates.bat that is called in the KB updates in the build task sequence. 
1. %SCRIPTROOT%\vcredist_x64.exe /q 
7) The psutil folder in the above zip file has been added to python files that are copied over during the build step.
8) Buildbot.bat has been replaced with the above file, https://bug781277.bugzilla.mozilla.org/attachment.cgi?id=779191
1. This file is already set up in a schedule task through GPO.

Comment 39

[Mark Cornmesser [:markco]]

Added MAPI header step to the build task sequence: 
cmd /c xcopy "z:\files\MAPI headers\*.*" "C:\Office 2010 Developer Resources\Outlook 2010 MAPI Headers\" (https://bugzilla.mozilla.org/show_bug.cgi?id=891956)

Recapturing the image and then will test with the deploy task sequence.

Comment 40

[Mark Cornmesser [:markco]]

jhopkins: I just kicked off installs on the test machines. They should now include more complete Python27 directories. The two files you mentioned twistd.bat and twistd.py were in a different Python directory, so I copied those files into the Python27 directory. Let me know if that worked, please.

Comment 41

[John Hopkins (:jhopkins)]

1) Take new c:\mozilla-build\runslave.py into the image/GPO

2) $HOME/.hgrc is misnamed ".hrgc"

3) Missing files under C:\mozilla-build\buildbotve, including scripts\activate.bat

On an existing w64 build slave, running "activate.bat", then "pip freeze" says:

Twisted==10.2.0
buildbot==0.8.4-pre-moz2
buildbot-slave==0.8.4-pre-moz2
distribute==0.6.14
pywin32==214
simplejson==2.1.3
zope.interface==3.6.1

Comment 42

[Mark Cornmesser [:markco]]

1) The file has been updated and the change will be picked up by GPO. 

2)Spelling error has been corrected in the source files for the GPO. Will be corrected by GPO> 

3)Diving deeper to find out what happened with those files during the install process.

Comment 43

[Mark Cornmesser [:markco]]

3)During the buildbot install task sequence step either directories are being over-written or just deleted. I will need to rework the buildbot.bat for 2008.I'll up date the ticket once I have a new install.

Comment 44

[Mark Cornmesser [:markco]]

The C:\mozilla-build\buildbotve\Scripts is now there.

C:\mozilla-build\buildbotve\scripts>C:\mozilla-build\buildbotve\scripts\activate
.bat
(buildbotve) C:\mozilla-build\buildbotve\scripts>pip.exe freeze
Twisted==10.2.0
buildbot==0.8.4-pre-moz2
buildbot-slave==0.8.4-pre-moz2
distribute==0.6.14
pywin32==214
simplejson==2.1.3


The 2 test machines have been re-imaged and are ready for testing. 

The reason it is taking so long between these iterations is because these files are captured within the wim. In the future these files will be handled Puppet eliminated the need to capture them within the wim.  


zope.interface==3.6.1

Comment 45

[John Hopkins (:jhopkins)]

Thanks, Mark!  Things are looking much better now.  You may need to verify the VNC password is correct - I had to use an older password earlier today but the latest password works now.
I think we're done the bulk of the configuration work and can start to focus our efforts on getting this into AWS.  There may be a configuration tweak or two along the way but no need for us to block on getting it perfect.

Comment 46

[John Hopkins (:jhopkins)]

Running into intermittent conftest.exe "permission denied" failures while doing a WINNT 5.2 mozilla-inbound pgo-build:

======
checking whether the C++ compiler (cl  ) works... no
------ config.log ------
rm: cannot lstat `conftest.exe': Permission denied
configure: error: installation or configuration problem: C++ compiler cannot create executables.
======

A possible solution (requiring GPO changes) is described at https://bugzilla.mozilla.org/show_bug.cgi?id=545015#c24  Should do more investigation as to what differs between production w64-ix and the new ix-mn build slaves that would cause this intermittent problem.

Comment 47

[John Hopkins (:jhopkins)]

Link to mozilla.dev.platform post initial cutover plan: https://groups.google.com/forum/#!topic/mozilla.dev.platform/zACrUe_JwKw

We hit a blocker this week in bug 925894 which has been resolved.

I will be sending out an updated win64-rev2 migration proposal today to Sheriffs, QA, A-Team, RelEng, and RelOps.

Comment 48

[John Hopkins (:jhopkins)]

Currently working through staging mozilla-beta l10n repack timeouts in bug 928982.

As proposed on the forum, we will start with two rev2 build machines mixed into the rev1 try pool and let that build for a ~week, then mozilla-central+inbound for a week, then look at aurora/beta/release/esr.

I don't expect larger scale machine reimaging to happen for another 2 weeks or so.

Comment 49

[John Hopkins (:jhopkins)]

Waiting until Monday to deploy two win64-rev2 build slaves onto try and try-comm-central so as not to bitrot an unrelated, complex uplift patch scheduled to land on Monday (per rail).

Comment 50

[John Hopkins (:jhopkins)]

Patch for mixed win64 rev1/rev2 pool on 'try' (bug 930506) landed and reconfigured.

Comment 51

[John Hopkins (:jhopkins)]

Mixed win64 rev1/rev2 try pool is enabled and the first rev2-based try build is running.

Comment 52

[John Hopkins (:jhopkins)]

No issues found with the Try builds.  :philor was happy with the test results as well.

mozilla-inbound and b2g-inbound branches will be switched to a mixed rev1/rev2 pool later today.

Comment 53

[John Hopkins (:jhopkins)]

Next week, we plan to migrate 40 each of win64-rev1 try build and production build slaves to win64-rev2.

Timetable:

Monday:
- jhopkins will buildbot-configs patch, split hosts in etherpad[1] into try and production batches, and check builds run over the weekend
- markco will verify hosts that need a reboot for ipmi
- jhopkins will disable hosts in off-peak hours in advance of the reimage to ensure they are idle by the time of reimaging

Tuesday to Friday:
- 7:00am Pacific: markco will ping jhopkins before he starts reimaging, then kick off reimages for 5 of each type 
- 12:00pm Pacific: same procedure as 7:00am, with 5 different slaves of each type

Note: RelEng will not be blocked on relops if builds do not finish in time since they can also reimage machines as needed.

[1] https://etherpad.mozilla.org/win64-rev2

Comment 54

[John Hopkins (:jhopkins)]

This week's cutovers are being tracked in bug 934976.  So far, things are going very well.

Comment 55

[John Hopkins (:jhopkins)]

The migration of 80 rev1 w64-ix-slave to rev2 (bug 934976) is complete.

I'll follow up with next steps on Monday.

Comment 56

[John Hopkins (:jhopkins)]

Planning to switch mozilla-central and comm-central to a mixed win64-rev1 + win64-rev2 pool tomorrow, per https://groups.google.com/forum/#!msg/mozilla.dev.platform/VSgwbVIuDwg/2a3y3gGq0OIJ

Comment 57

[John Hopkins (:jhopkins)]

try/try-comm-central are now building only on win64-rev2.  Builds are looking good.

mozilla-central and comm-central are configured to use a mixed pool of win64-rev1 and win64-rev2 build machines.  m-c is currently closed for other reasons, so it will take longer to verify that things are working as expected.

Comment 58

[John Hopkins (:jhopkins)]

Updated status:

* mozilla-central and comm-central branches are building only on win64-rev2
* mozilla-inbound and b2g-inbound remain on a mixed rev1/rev2 pool to ensure an adequate number of build slaves to meet its high checkin load
* 24 win64-rev1 build machines are left to reimage, pending cutover of aurora, beta, release, esr branches.
* Firefox desktop nightlies are building on win64-rev2 since Tuesday, November 27

What's next:

* start building aurora on win64-rev2 in 1 week
* start building beta on win64-rev2 for beta 1

Comment 59

[Amy Rich [:arr] [:arich]]

Now that we've verified that the new imaging/management system is working, this goal is complete.  Please open separate bugs to handle additional machine reimagings.

Comment 60

[John Hopkins (:jhopkins)]

Note: Bug 918414 will be used to track the remaining Release Engineering work.