Closed Bug 780050 Opened 12 years ago Closed 11 years ago

(Wellington / Addison) created MDT-driven automated install configuration for windows 8 test machines

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Platform:
x86_64
Windows 8
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: armenzg, Assigned: q)

References

(
URL
)

Details

(Whiteboard: [2013Q1])

Attachments

(2 files, 1 obsolete file)

basedir for win8 64-bit machines
1.52 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
ss.jpg
12.25 KB, image/jpeg
Details 
If any of the steps gives you trouble you can reach coop to help figure them out.
Some files will need an official home like the appache configuration file or the bat files that I grab from people.

You can use slaves from bug 780008 to experiment with this (since the multi ix node is the one I use):
10.12.40.71
10.12.40.72
10.12.40.73

You can use the win8 RC or wait until Aug. 15th for the final win8 version.

Thanks!

* enable quick edit mode (cmd as normal user as well as run as Administrator)
REG ADD "HKEY_CURRENT_USER\Console" /v QuickEdit /t REG_DWORD /d 1 /f
Open cmd as administrator and type 'lusrmgr'
* Users -> New User...
** cltbld;
Modify the properties of it  
* "User cannot change password" (uncheck), "password never expires" (checked)
* Add "Remote Desktop Users" to "Member Of" 
Allow cltbld to restart the system  
* gpedit.msc (as administrator) 
* Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> User Rights Assignment  
Double click 'Shut down the system', add cltbld to the list.  
* run cmd as Administrator and type:
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultUserName /d cltbld /t REG_SZ /f
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultPassword /d newpasswd /t REG_SZ /f
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v AutoAdminLogin /d 1 /t REG_SZ /f
* Install http://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-1.6.exe
* wget -OC:\\Users\\cltbld\\Desktop\\install-buildbot.bat http://people.mozilla.com/~armenzg/incoming/install_buildbot.bat
* mkdir /c/talos-slave
* Append ';C:\mozilla-build\python;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\buildbotve\scripts' to system path.
* cmd as Administrator
REG ADD "HKLM\Software\Wow6432Node\Python\PythonCore\2.7\InstallPath" /t REG_SZ /d "C:\mozilla-build\python" /f
wget http://downloads.sourceforge.net/project/pywin32/pywin32/Build%20217/pywin32-217.win32-py2.7.exe?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fpywin32%2Ffiles%2Fpywin32%2FBuild%2520217%2F&ts=1341502402&use_mirror=hivelocity
wget -OC:\\Users\\cltbld\\Desktop\\runslave.py http://hg.mozilla.org/build/puppet-manifests/raw-file/9976e0832434/modules/buildslave/files/runslave.py
wget -OC:\\Users\\cltbld\\Desktop\\startTalos.bat http://people.mozilla.com/~armenzg/incoming/startTalos-win8.bat
schtasks /create /tn talosslave /tr "C:\Users\cltbld\Desktop\startTalos.bat" /sc onlogon  /ru cltbld
* Install gVim 7.3.46
wget -OC:\\Users\\cltbld\\Desktop\\gvim73_46.exe ftp://ftp.vim.org/pub/vim/pc/gvim73_46.exe
* Install Apache 2.2.22
wget -OC:\\Users\\cltbld\\Desktop\\httpd-2.2.22-win32-x86-no_ssl.msi http://apache.mirror.nexicom.net//httpd/binaries/win32/httpd-2.2.22-win32-x86-no_ssl.msi
* Set 'Network Domain' and 'Server Name' to 'localhost'. 
* Create an empty directory so Apache doesn't complain: 
mkdir c:/talos-slave/talos-data/talos
* Edit C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf\httpd.conf
* Set "DocumentRoot" to "C:\talos-slave\talos-data\talos"
* Comment any lines that begin with "CustomLog"
** Run "startTalos.bat" manually as Administrator so you see the Firewall prompts and add the exceptions
Assignee: server-ops-releng → mlarrain
Summary: Automate these steps to setup a win8 machine → created MDT-driven automated install configuration for windows 8 test machines
Since Release Preview came out I had to upgrade MDT which I did today. As well I created a deployment share for Win8 exclusivity for testing of the new image. Created a task sequence for a base image to add some registry settings for the deployment. During the testing of pushing out a base image it doesn't finish all the way to the desktop so doing some research as to why this isnt working to completion (It is stopping asking for me to create a username and password even though I have already configured the admin account with a password).
yay it works now after upgrading MDT to the latest version and creating a new deployment share for it. I presume that it has to do with windows 8 not liking the custom rules (This was something that was brought up with the latest MDT changes). Working on the registry settings and other steps that Armen listed above.
Switched over to the latest version of Windows 8 for automated deployment and working on the rest of the config files.
file is ready
I was able to get Windows 8 RTM to be auto deployed via MDT as well as the auto logon and switch to desktop.
Writing up process list for discussion on what to use MDT for vs what to use GPO for. Meeting scheduled with Dustin for tomorrow to review and update documentation. This is for both builders and testers.
at the moment I am stuck at pushing the win8 image out with MDT to the ix multinode box. The process seems to work up until the very end where it locks the machine up completely. Working on troubleshooting these issues so we can confirm that this system will work for us.
I was able to get windows 8 deployed to the iX multinode now without issues. It now deploys with MDT and allows for RDP.
Was able to emulate all of this using GPO today

REG ADD "HKEY_CURRENT_USER\Console" /v QuickEdit /t REG_DWORD /d 1 /f
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultUserName /d cltbld /t REG_SZ /f
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v DefaultPassword /d newpasswd /t REG_SZ /f
reg add "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon" /v AutoAdminLogin /d 1 /t REG_SZ /f
#reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon"  /v AutoLogonCount

Also added rights for cltbld account to be able to shutdown the system as well as RDP access.

Also wrote two scripts for installing buildbot tools;

xcopy /d /e /y \\releng.ad.mozilla.com\SysVol\releng.ad.mozilla.com\Policies\{FF3916B2-636A-4A78-B2D0-B4564967677A}\User\Scripts\Logon\buildbot\*.* C:\mozilla-build

and appending the environment path;

strComputer = "."

Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")

Set colItems = objWMIService.ExecQuery _
    ("Select * From Win32_Environment Where Name = 'Path'")

For Each objItem in colItems
    strPath = objItem.VariableValue & ";C:\mozilla-build\python;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\buildbotve\scripts"
    objItem.VariableValue = strPath
    objItem.Put_
Next
> xcopy /d /e /y \\releng.ad.mozilla.com\SysVol\releng.ad.mozilla.com\Policies\{FF3916B2-636A-4A78-B2D0-B4564967677A}\User\Scripts\Logon\buildbot\*.* C:\mozilla-build

This concerns me, because buildbot != mozilla-build.  So either this is installing the wrong thing, or it's very confusingly misnamed.

How confident are you that this task seq / GPO will work for win8-32 (Addison)?
Summary: created MDT-driven automated install configuration for windows 8 test machines → (Wellington / Addison) created MDT-driven automated install configuration for windows 8 test machines
this is taking mozilla-build1.6.exe and extracting it to a folder on the sysvol share and then copying that folder to C:\mozilla-build
Right, why's it copying it from a folder named "buildbot"?
cause that is just what I happened to call it I can change it
folder name has been changed on sysvol to mozilla-build
I need a copy of CopSSH for the Windows 8 Test slave
Ran into an issue with getting some commands to run:
wget -OC:\\Users\\cltbld\\Desktop\\install-buildbot.bat http://people.mozilla.com/~armenzg/incoming/install_buildbot.bat

I fixed the issues by running the command as: 

wget -OC:\\Users\\cltbld\\Desktop\\install-buildbot.bat "http://people.mozilla.com/~armenzg/incoming/install_buildbot.bat"
Depends on: 792456
Converted most of the applications in this build to msi's for ease in installing these without needing hands on assistance. Still to convert is the Apache installer, gvim and ultraVNC. which should be fairly straight forwards to convert.
After a long time of changes and modifications I have been able to build the Windows 8 tester image in MDT (minus a ssh agent). This included fixing/figuring out how to; Enabling RDP which required opening ports and also turning on the RDP service, this is handled with a script that is run during the MDT process. Appending the Environment path so it stays persistent, I was able to achieve this by running a powershell script during the deploy. All the applications (Mozilla Build 1.6, pywin, Apache, gvim, etc..) were all installed via vb scripts. Moving startTalos-win8.bat and showDesktop.scf was easier said then done, I had to move the files to the root of C:\ and then add the scheduled task. This step here doesn't apply on first login and requires a reboot after the image is finished however it does work consistently as I have tested this multiple times. The final step was to figure out how to change registry keys that allow for automatic login of cltbld. These keys are also being used by MDT to do the image deploy so it required a modification of one of MDT scripts. I have thus captured a thick image of this deploy and pushed a new deploy with it and returned with success. Next is to give this image to release engineering for testing.
Status: NEW → ASSIGNED
Armen I was testing the deploy and realized that apache wasn't starting properly. Turns out when I run the command:

"C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin\
httpd.exe" -w -n "Apache2.2" -k start

I get this error:

[Tue Oct 16 17:06:45 2012] [error] (OS 2)The system cannot find the file specifi
ed.  : No installed service named "Apache2.2".
Note the errors or messages above, and press the <ESC> key to exit.  28...

Please advise if you had the same issue and or troubleshooting steps you'd recommend. This was installed with the command line flags of the MSI you suggested in the etherpad.
Sorry. I have tried reproducing this on the machine I got but I can't.

Who is starting that line?
Is it starting as Administrator?

If I can get access to that machine and steps to reproduce I can give you a hand.
I will push a deploy to a machine for you to see and test with so we can figure out why this is happening.
It seems that Apache did not get installed as a service.

To install it as a service do this (or re-install it from scratch):
"C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin\httpd.exe" -k install [1]

It will prompt you to add a Firewall exception http://cl.ly/KHdc

After that you can try again:
"C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin\httpd.exe" -w -n "Apache2.2" -k start


[1] 
http://httpd.apache.org/docs/2.2/platform/windows.html
I will make these changes into the deployment and test again.
How did it go?
You can re-deploy to that instance on SCL you got me access to on Friday.

IIRC you said that you were:
* going to add the firewall rule in advance
* install Apache after that

Let me know if it does not go well and if I can check again.
tested some steps and might be able to work around it with a script called LTISuspend.wsf that basically pauses the ref image deploy for us to do any manual steps and then you just double click a desktop icon to continue the MDT deploy. This will only be done during a new ref image capture however. 

Armen and I are looking into a second error that apache is saying there is no email address in place on the httpd.conf file.
Was able to get the firewall issue taken care of by using the LTISuspend.wsf script. Now it's prompting saying port 80 is already in use. Per using netcat I was able to verify apache was already running. Even on reboot apache seems to be running and the start-talos script issues a warning that it cant bind to 0.0.0.0:80 as it is in use already. Armen thoughts on this?
I replied by email.

I think that the schedule task is not running with the highest privilege and/or cltbld had not been created as part of the Administrators group.
it was discussed long ago that cltbld would not be part of the administrators group iirc. What other steps can we take to solve this issue?
Hi MaRu,
I noticed that you were looking into allowing Apache on the Firewall but I just remembered that we keep the Firewall disabled on the Windows 7 and XP machines.

Should we do that for Windows 8 as well?

We also have to prevent Windows Defender from running or getting on the way.
I have adjusted my steps for Windows 8 32-bit:
https://etherpad.mozilla.org/armenzg-win8-32-bit

I think the only differences is:
* updated httpd.conf
* new startTalos.bat
* the Apache logs now go to C:\talos-slave\{error,acess}.log (which is why the first two changes)
* a new way of creating the scheduled task [1]
* add a task to go directly into Desktop mode [2]

We also need to disable Windows Defender.

I will try to run my slave through staging and see where we are.
MaRu is also going to create a slave in an automated way for me to put on staging.

[1]
wget -OC:\\Users\\cltbld\\Desktop\\talosslave.xml "http://people.mozilla.com/~armenzg/incoming/talosslave.xml"
schtasks /create /tn talosslave /xml "C:\Users\cltbld\Desktop\talosslave.xml"
[2]
wget -OC:\\Users\\cltbld\\Desktop\\showDesktop.scf  "http://people.mozilla.com/~armenzg/incoming/showDesktop.scf"
schtasks /create /tn ShowDesktop /tr "C:\Users\cltbld\Desktop\showDesktop.scf" /sc onlogon  /ru cltbld
    Armen I pushed what we have to ix-mn-w764-001.wintest.releng.scl3.mozilla.com please verify
Please change the hostname of this machine and it's OOB interface so as to avoid confusion.
Changing hostname and DNS now to ix-mn-w864-001.wintest.releng.scl3.mozilla.com
DNS and inventory have been updated.
I got my machine to connect to buildbot but I have hit some weirdness that I'm trying to debug (more tomorrow):

 (view as text)

'bash' '-c' 'pwd'
 in dir C:\talos-slave\test\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'pwd']
 environment:
  ALLUSERSPROFILE=C:\ProgramData
  APPDATA=C:\Users\cltbld\AppData\Roaming
  COMMONPROGRAMFILES=C:\Program Files\Common Files
  COMPUTERNAME=IX-MN-W832-001
  COMSPEC=C:\windows\system32\cmd.exe
  FP_NO_HOST_CHECK=NO
  HOMEDRIVE=C:
  HOMEPATH=\Users\cltbld
  LOCALAPPDATA=C:\Users\cltbld\AppData\Local
  LOGONSERVER=\\IX-MN-W832-001
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_NO_REMOTE=1
  NO_EM_RESTART=1
  NUMBER_OF_PROCESSORS=8
  OS=Windows_NT
  PATH=C:\mozilla-build;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;C:\mozilla-build\buildbotve\scripts;C:\mozilla-build\python;C:\mozilla-build\python\Scripts;C:\mozilla-build\hg;C:\mozilla-build\7zip;C:\mozilla-build\upx203w;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32;C:\WINDOWS;
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_IDENTIFIER=x86 Family 6 Model 30 Stepping 5, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=1e05
  PROGRAMDATA=C:\ProgramData
  PROGRAMFILES=C:\Program Files
  PROMPT=$P$G
  PSMODULEPATH=C:\windows\system32\WindowsPowerShell\v1.0\Modules\
  PUBLIC=C:\Users\Public
  PWD=C:\talos-slave\test
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:\windows
  TEMP=C:\Users\cltbld\AppData\Local\Temp
  TMP=C:\Users\cltbld\AppData\Local\Temp
  USERDOMAIN=IX-MN-W832-001
  USERDOMAIN_ROAMINGPROFILE=IX-MN-W832-001
  USERNAME=cltbld
  USERPROFILE=C:\Users\cltbld
  WINDIR=C:\windows
  XPCOM_DEBUG_BREAK=warn
 using PTY: False
error in RunProcess._startCommand
Traceback (most recent call last):
  File "C:\mozilla-build\buildbotve\lib\site-packages\buildslave\runprocess.py", line 377, in start
    self._startCommand()
  File "C:\mozilla-build\buildbotve\lib\site-packages\buildslave\runprocess.py", line 497, in _startCommand
    usePTY=self.usePTY)
  File "C:\mozilla-build\buildbotve\lib\site-packages\buildslave\runprocess.py", line 523, in _spawnProcess
    path, usePTY=usePTY)
  File "C:\mozilla-build\buildbotve\lib\site-packages\twisted\internet\posixbase.py", line 335, in spawnProcess
    raise NotImplementedError, "spawnProcess not available since pywin32 is not installed."
NotImplementedError: spawnProcess not available since pywin32 is not installed.

remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
Failure: buildslave.exceptions.AbandonChain: -1
]
ix-mn-w832-001 (which I setup manually) is having problems to set pywin32api:
C:\Users\Administrator>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed: The specified module could not be found.

Unfortunately we need this which contains win32process which buildbot uses.

ix-mn-w864-001 (which was setup automatically by MaRu) does not have the problem even though we followed the same steps.

I will keep on debugging.

If anyone has a Windows 8 *32-bit* machine at hand please try to do this:
wget -OC:\\%HOMEPATH%\\Desktop\\MozillaBuildSetup-1.6.exe http://ftp.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/MozillaBuildSetup-1.6.exe
wget -OC:\\%HOMEPATH%\\Desktop\\pywin32-217.win32-py2.7.exe "http://downloads.sourceforge.net/project/pywin32/pywin32/Build%20217/pywin32-217.win32-py2.7.exe?r=&ts=1346358482&use_mirror=iweb"
C:\\%HOMEPATH%\\Desktop\\MozillaBuildSetup-1.6.exe
REM run the next step by opening cmd as Administrator
REG ADD "HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath" /t REG_SZ /d "C:\mozilla-build\python" /f
C:\\%HOMEPATH%\\Desktop\\pywin32-217.win32-py2.7.exe
Add mhammond as he is the man, I hear.

I'm now trying directly with Python for Windows + pywin32 without using mozilla-build:
http://www.python.org/ftp/python/2.7.3/python-2.7.3.msi
It seems that installing Python from python.org does the job.
Let me see if I can get buildbot working with this.
I ended up doing this (after uninstalling all python, mozilla-build and registry key changes for Python:
* Install mozilla-build 1.6
* Install python2.7 to C:\mozilla-build\python27
* Install pywin32 to C:\mozilla-build\python27
* Modify install_buildbot.bat to:
** SET pythondir=%mozillabuild%\python27\
** SET pythondll=python27.dll
** COPY %mozillabuild%\python\\%pythondll% %virtualenv%\scripts 
*** Python2.7 does not come with the dll

At this point I can run this successfully:
C:\windows\system32>C:\mozilla-build\buildbotve\scripts\python.exe
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32process
>>>

I'm going to put the slave on staging.

[1] http://people.mozilla.com/~armenzg/win8/install_buildbot.bat
I have filed bug 805627 to deal with the MozillaBuild + pywin32 + Windows 8 32-bit issue.

I'm going to direct Windows 8 32-bit status updates to bug 803595 where it should have been going from the beginning.

I will test MaRu's ix-mn-w864-001 tomorrow (if I don't take PTO after all).
Hi MaRu,
Here's my first evaluation:
* The machine requests to be activated - needs to be fixed
* C:\startTalos-win8.bat should match this [1]
** FYI I can see at the beginning of each line there is a \t
** we're trying to move away from using Apache
** C:\runslave.py instead of C:\Users\cltbld\Desktop\runslave.py
* Use this httpd.conf [2]
** The A-team and I are working towards not needing Apache at all though
* I found that we need CoreUtils which I installed to C:\CoreUtils:

Can you also update this?
* Append  ';C:\mozilla-build\python;C:\mozilla-build\python\Scripts;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\info-zip;C:\CoreUtils\bin;C:\mozilla-build\buildbotve\scripts'  to system path.

After these changes I've managed to connect the slave to buildbot. 

Should we move runslave.py, showDesktop and startTalos-win8.bat to C:\talos-slave since we know it always exists and does not require Administrator privileges?

On another note, would you mind naming startTalos-win8.bat as starTalos.bat to keep the consistency across OSes?

[1] http://people.mozilla.com/~armenzg/win8/startTalos-win8-64.bat
[2] http://people.mozilla.com/~armenzg/win8/httpd.conf
I have changed the env. path and yes I agree we should use C:\talos-slave 
I have also renamed startTalos-win8 to startTalos.bat
Hi MaRu,
The scheduled task is trying to call C:\startTalos.bat rather than C:\talos-slave\startTalos.bat.

The startTalos.bat has to call C:\talos-slave\runslave.py.
You also need to remove lines 6-10 which deal with Apache. It should match [1].

I also don't see the newer httpd.conf being used. It should match [2]

An easy way yo see if the machine is working is by imaging it, rebooting it and check that on cltbld there is a command prompt running.

[1] http://people.mozilla.com/~armenzg/win8/startTalos-win8-64.bat
[2] http://people.mozilla.com/~armenzg/win8/httpd.conf
Thanks Armen I will fix this right now and repush that image later tonight.
The latest pushed image calls to C:\Users\cltbld\Desktop\runslave.py rather than C:\slave\runslave.py.

I also need the hostname to be ix-mn-w864-001 otherwise I will get a 404 when calling slavealloc:
http://slavealloc.build.mozilla.org/gettac/ix-mn-w864-001
Apache seems to be working as of now. Next step is to fix talos-slave.bat it is failing to run the last command; C:\mozilla-build\python\python c:\slave\runslave.py
I updated the slave to match C:\slave
 update slaves set basedir='C:\\slave' where name like 'ix-mn-w864-001';

I have also updated my etherpad and scripts on people:
https://etherpad.mozilla.org/armenzg-win8
Yesterday armen and I were able to fix VNC to run as a service at boot and allow him to login. At that point we were able to confirm that this image was ready to be captured(though we are still missing an ssh client which armen is working on qualifying). After capturing the image and trying to deploy it and join a domain a few of the features stopped working such as the showDesktop.wsf script. I am working on trying to fix this with registry changes instead of relying on a script to run with scheduled tasks.
Attached patch basedir for win8 64-bit machines (obsolete) — — Splinter Review 

        Attachment #678416 -
        Flags: review?(coop)

    
    

    
  
Comment on attachment 678416 [details] [diff] [review]
basedir for win8 64-bit machines

Review of attachment 678416 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/buildslave/files/runslave.py
@@ +417,5 @@
>      else:
>          # All POSIX slaves are consistent about the location.  Woo!
>          return [ '/tools/buildbot/bin/twistd' ]
>  
> +def test_enough_screen_resolution():

test_enough_screen_resolution() rankles as a function name. Could we go with verify_screen_resolution() or similar?

@@ +504,5 @@
>          if not options.no_start:
>              notif = NSCANotifier(options)
>              notif.try_send_notice()
> +            if not test_enough_screen_resolution()
> +                print >>sys.stderr, "FATAL: We don't have a dongle attach"

sp.: attached

        Attachment #678416 -
        Flags: review?(coop) → review+

    
    

    
  


  
I had junk on the previous patch. I removed the dirt.

        Attachment #678459 -
        Flags: review?(coop)

        Attachment #678459 -
        Flags: review?(coop) → review+

    
    

    
  
Please remember to update the copy in build/puppet as well:
  http://hg.mozilla.org/build/puppet/file/tip/modules/buildslave/files/runslave.py
Coop's r+ is fine for that update.

    
  

        Attachment #678416 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 678459 [details] [diff] [review]
basedir for win8 64-bit machines

2d1b0425711e

        Attachment #678459 -
        Flags: checked-in+

    
    

    
  
I have updated the etherpad to use this:
wget -OC:\\slave\\runslave.py "http://hg.mozilla.org/build/puppet-manifests/raw-file/2d1b0425711e/modules/buildslave/files/runslave.py"

I have tested that it works.

    
    

    
  
(In reply to Dustin J. Mitchell [:dustin] from comment #52)
> Please remember to update the copy in build/puppet as well:
>  
> http://hg.mozilla.org/build/puppet/file/tip/modules/buildslave/files/
> runslave.py
> Coop's r+ is fine for that update.

*nudge*

    
    

    
  
My bad. I wasn't aware that we had two repos:
http://hg.mozilla.org/build/puppet/rev/dcbc47cbe48d

    
    

    
  
MaRu, we also have to add exceptions for Python and ssltunel for test jobs to work.

I never knew of this because I probably tested Windows 8 without firewall at that time.

[1] http://cl.ly/KkAp
[2] http://cl.ly/Kk11

    
    

    
  
I will add those exceptions now then.

    
    

    
  
Hi MaRu,
We also have to add this to the imaging process:
* Install Microsoft VC10 Debug CRT
wget %USERPROFILE%\\Desktop\\Microsoft_VC100_DebugCRT_x86.msi http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/Microsoft_VC100_DebugCRT_x86.msi

I that when I tested win8 few months ago I did not go as deep into the results as I'm this time around. I think I just made sure that we could run buildbot and that optimized jobs were running and that there was no problem on changing the screen resolution. I don't think I run debug jobs or talos jobs.

This is the only I can explain that I'm finding all these things now.
I know that there are some changes on how the machine got setup manually VS automatically but not all of them are due to that.

I'm sorry again and I hope to find things faster now.

    
    

    
  
not a problem I am going to put this into the task sequence and test it on ix-mn-w864-001 if that is fine with you.

    
    

    
  
Let's do this:
ix-mn-w864-001 is for you and
ix-mn-w864-002 is for me!

    
    

    
  
Hi MaRu,
I did not find CoreUtils installed on ix-mn-w864-002.
Can you please add it to the task sequence and test it on 001?

It should be installed to C:\CoreUtils.
http://softlayer.dl.sourceforge.net/project/gnuwin32/coreutils/5.3.0/coreutils-5.3.0.exe

    
    

    
  
yeah i need to get it on 001 today was working on making it a task sequence yesterday for you but didn't finish it yet. As soon as I have it on there I will let you test the box out :)
Severity: normal → critical

    
    

    
  
FTR we will most likely have to add Python25 (+pywin32) and go back to use C:\talos-slave rather than C:\slave so it works with mozharness unit tests which will be here very soon (it is currently running on Cedar).

    
    

    
  
I just got state on the current Windows 8 x68 MDT deploy, so I think I can take over here, as best as possible.

I don't like to hear about going back to Python2.5 or an old directory name, but hey, I don't have to like it.
Assignee: mlarrain → dustin

    
    

    
  
I won't have time to work on this for a couple of weeks.
We can get Tomcat to help but we might be able to not need this by adding pywin32 to WinXP and Win7 slaves and move them to Python2.7.
I can give instructions on how to do it.
We can file a releng bug to fix that.

The other alternative is to have slight different configs for ix machines and use the current python directories (probably best way).
Assignee: dustin → server-ops-releng
Whiteboard: [2013Q1]
Assignee: server-ops-releng → qfortier
Assignee: qfortier → cnackers

    
    

    
  
Chris is going to wrap this up.  The major blocking issue is solving the graphics-card problem.  Ideally we'll be able to boot on the onboard graphics, but make sure Firefox runs on the add-on card.

Chris is starting with ix-mn-w864-001, which has its onboard graphics enabled right now.  Armen is done with ix-mn-w864-002, so we can use that too, but we'll need to get hands-on to restore the onboard graphics first.  I've updated notes in inventory with this status.

The point of contact in releng for further testing is tomcat.

    
    

    
  
Quick notes on the dual graphics cards hacks.
(still being tested; this should eventually be replaced by a cleaner TS or two VBS scripts)  :

    * Detect new display, Add “Try Monitor on VGA”
    * Set Monitor to 1600 x 1200
    *Add login script ( currently hacked into starttoas login script) to run displayswitch /internal ( don’t ask me why windows makes the extra non mb card internal)
    *Add logout script to return to displayswitch /external (back to the on board card) (NOT NECESSARY)
    *Notes:
        ** This allows the inband card to remain active
        ** During the time that the nvidia addon card is in use the console redirect doesn’t work but VNC does
        ** The video will always revert to the onboard card during boot
        ** (LATER) Need to come up with a remote (powershell, wmi command, etc) way of triggering the display back in case VNC fails.

    
    

    
  
A short list of things to check/do from getting this host set up.  Some of these are in the google doc, and some are new.

 * allow ping through the host firewall
 * add to the domain
 * disable updates (for now; we'll need to get WSUS set up with a fixed group of updates before putting this in production)

    
    

    
  
Expanded and (to our knowledge) comprehensive task list, as agreed in a vidyo meeting just now:

 * Blocker
   * Verify
     * does not address sound drivers (fixed - Virtual Audio Driver; dustin to verify and document, fix if necessary)
     * CoreUtils (Dustin to check if already installed, work with Chris if not)
     * Buildbot  startup process runs from sch tasks w/ elevated priviledge  (Dustin to  verify if in place, with armen for details)
     * Firewall exceptions for Python (Dustin to verify if in place, work with armen to verify, document)
     * Firewall exceptions for ssltunnel (Dustin to verify if in place, work with armen to verify, document)
   * Change
     * does not add host to domain - Q (domain/GPO setup) and Chris (MDT)
     * does not  install "Mozilla Maintenance Service" - allows testing of updates -  releng for more information (armen) (registry keys, etc, armen has more  info)
     * does not  handle secondary graphics card - (Q and Chris; dustin to get accurate  state) (mostly or completely implemented, but needs testing from releng)
     * automatic updates should be disabled, connected to WSUS (Chris)
 * Non-Blocker
   * Verify
     * VNC Configuration that doesn't suck (Dustin to look, work with Q to make any improvements)
   * Change
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
 * any other issues discovered in testing (unlikely to be an empty set, given state loss from Matt's departure)

This is sorted into blockers and non-blockers, and subsequently into things we think are done but need to check, and things we know are not done.  Time scales aren't known yet, without input from the affected parties.

    
    

    
  
(ask dustin for access to the spec google doc in URL)
URL: https://docs.google.com/document/d/1g...

    
    

    
  
> * Blocker
>   * Verify
>     * does not address sound drivers (fixed - Virtual Audio Driver; dustin to verify and document, fix if necessary)

Not installed - needs to be fixed and documented

>     * CoreUtils (Dustin to check if already installed, work with Chris if not)

Installed

>     * Buildbot  startup process runs from sch tasks w/ elevated priviledge  (Dustin to  verify if in place, with armen for details)

Not the case - needs to be fixed and documented

>     * Firewall exceptions for Python (Dustin to verify if in place, work with armen to verify, document)
>     * Firewall exceptions for ssltunnel (Dustin to verify if in place, work with armen to verify, document)

In place.

>     * VNC Configuration that doesn't suck (Dustin to look, work with Q to make any improvements)

VNC seems perfectly fine to me (and I recall the pain of w764 clearly).

I discovered some other minor issues, detailed below (see bitlocker, autologincount, activation, other scheduled tasks, admin password)

----

So the updated task list is

 * Blocker
   * Change
     * Buildbot scheduled task does not run w/ highest priviledges (+ documentation)
     * install sound drivers (Virtual Audio Driver) + documentation (Chris)
     * does not add host to domain - Q (domain/GPO setup) and Chris (MDT)
     * does not install "Mozilla Maintenance Service" - allows testing of updates -  releng for more information (armen) (registry keys, etc, armen has more  info)
     * does not handle secondary graphics card - (Q and Chris) - doing this by hand now, will automate
     * automatic updates should be disabled, or connected to WSUS (Chris)
     * BitLocker should be disabled - already the case in practice, but the TS tries to enable it (Chris)
     * AutoLogin needs to occur >999 times (simple registry change; Dustin)
     * Set local administrator password to a secure password (Dustin)
     * Windows Activation (might "just work" when added to the domain; Q)
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
     * Disable other system-supplied scheduled tasks
 * any other issues discovered in testing (unlikely to be an empty set, given state loss from Matt's departure)

    
    

    
  
AutoLogin is not actually a problem, per Q and some experimentation.
I fixed the local administrator password.

    
    

    
  
Autologon should only quit working if you update the CLTBLD account password. The script has the password for that in it, so if you change that in the Task Sequence, that will break the script.

    
    

    
  
The scheduled task should also be configured to only run when cltbld logs in.

    
    

    
  
Bitlocker disabled in Task Sequence
Windows Updates disabled in image and Task Sequence

    
    

    
  
The scheduled task is fixed on -002, but not automated yet.

For adding hosts to the domain: OU's are set up for adding hosts to the domain, but still need some MDT setup - Q and Chris are working on that

I need a bit more detail from Armen on Mozilla Maintenance Service, and to talk to Chris about how to automate it.  I've looked at the docs for Win7, and they describe how to do it by hand, but not really what the result is supposed to be.

As for the secondary graphics card: it's configured by hand on all testing hosts (-001 .. -006).  Q has the config set up to enforce the monitor settings, but hasn't figured out how to add the monitor automatically.  He's talking to Chris.

----

Further updated task list, based on the above and a conversation with Q:

 * Blocker
   * Change
     * Buildbot scheduled task does not run w/ highest priviledges (+ documentation) (Chris)
     * install sound drivers (Virtual Audio Driver) + documentation (Chris)
     * does not add host to domain - Q (domain/GPO setup) and Chris (MDT)
     * does not install "Mozilla Maintenance Service" (armen)
     * does not handle secondary graphics card - (Q and Chris)
   * Verify
     * Windows Activation should "just work" when hosts are added to the domain
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
     * Disable other system-supplied scheduled tasks
 * any other issues discovered in testing (unlikely to be an empty set, given state loss from Matt's departure)

    
    

    
  
ix-mn-w864-001 - 006 all have the manual implementation of the "GPU display fix" allowing a larger display on the second card. The display will switch automatically when ctlbuild logs in before any of the python scripts are run. For the time being This will need to be setup again if the machines are re-imaged.

    
    

    
  

      
      
      
      

        Attached image
          
        ss.jpg
        — 
      

    


  
By the way, I verified the GPU's in use as follows:
 - install Firefox using IE
 - Go to "about:support"
 - Scroll down to "Graphics".  You want to see 1/1, meaning one of one windows is accelerated.

    
    

    
  
I've updated the google doc with everything I know and everything I could figure out, along with a bunch of comments.  Armen, Q, Chris, please have a look.

A few additions to the list:
 * Blocker
   * Verify
     * Verify that C:\mozilla-build\python is adequate (armen)
   * Change
     * Synchronous GPO Updates
     * Synchronous Startup / Login Scripts
       - both of these will be necessary to avoid polluting Talos numbers
     * Replace the current pywin install (with an undocumented MSI) with an Application
       - a simple change for Chris, and should have the same effect, but let's be sure
 * Non-Blocker
   * RDP should be disabled entirely (Chris)
   * disable indexing - bug 798578 (Q)
   * Determine which show-desktop method works (two are in place now) (Chris)
   * Buildbot install script should use a different source for packages than scl1-production-puppet (dustin)
   * Install GVim using an Application rather than an MSI

And removals:
 * Disable other system-supplied scheduled tasks
   - armen indicates we should leave these in place.

I'll revise the task list in comment 78 later today or early Monday.

    
    

    
  
Updated task list:

 * Blocker
   * Change
     * Buildbot scheduled task does not run w/ highest priviledges (+ documentation) (Chris)
     * install sound drivers (Virtual Audio Driver) + documentation (Chris)
     * does not add host to domain - Q (domain/GPO setup) and Chris (MDT)
     * does not install "Mozilla Maintenance Service" (armen)
     * does not handle secondary graphics card - (Q and Chris)
     * Synchronous GPO Updates (Q and Chris)
     * Synchronous Startup / Login Scripts (Q and Chris)
     * Replace the current pywin install (with an undocumented MSI) with an Application (Q and Chris)
   * Verify
     * Windows Activation should "just work" when hosts are added to the domain
     * Verify that C:\mozilla-build\python is adequate (armen)
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * RDP should be disabled entirely (Chris)
     * disable indexing - bug 798578 (Q)
     * Determine which show-desktop method works (two are in place now) (Chris)
     * Buildbot install script should use a different source for packages than scl1-production-puppet (dustin)
     * Install GVim using an Application rather than an MSI
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
 * any other issues discovered in testing

    
    

    
  
filing bugs that i notice on ix-mn-w864-002 depended on bug 836999.

Seems so far the elevated rights issue is the most blocking one - bug 840926 - running startTalos.bat as admin and running the tests then finish without any problem

    
    

    
  
to add so far:

 * Blocker
   * Change
     * Add C:\talos-slave\test\build\venv\scripts\python.exe firewall exception + docs (bug 840919) (Chris)
     * Change the path for the ssltunnel firewall exception (bug 840920) (Tomcat, Chris)

    
    

    
  
Corrections/additions to comment 84.  I'll post a complete update in a few hours.

  * Blocker
    * Change
      * Add firewall exception for C:\slave\test\build\venv\scripts\python.exe (bug 840919) (Chris)
      * Add firewall exception for C:\slave\test\build\tests\bin\ssltunnel.exe (bug 840920) (Chris)
      * Set up Mozharness virtualenv at c:\slave\test\build\venv (not c:\talos-slave) (Tomcat)
    * Investigate
      * Permissions problems (bug 840926) (Tomcat)

    
    

    
  
Firewall exceptions are added.  Indexing is disabled.  The blockers are in various states of progress with the usual nothing-is-easy complications.  Updated task list:

 * Blocker
   * Change
     * Buildbot scheduled task does not run w/ highest privileges (+ documentation) (Chris)
     * install sound drivers (Virtual Audio Driver) + documentation (Chris)
     * does not add host to domain - Q (domain/GPO setup) and Chris (MDT)
     * does not install "Mozilla Maintenance Service" (Chris)
     * does not handle secondary graphics card - (Q and Chris)
     * Synchronous GPO Updates (Q and Chris)
     * Synchronous Startup / Login Scripts (Q and Chris)
     * Replace the current pywin install (with an undocumented MSI) with an Application (Q and Chris)
     * Set up Mozharness virtualenv at c:\slave\test\build\venv (not c:\talos-slave) (Tomcat)
   * Verify
     * Windows Activation should "just work" when hosts are added to the domain
     * Verify that C:\mozilla-build\python is adequate (armen)
    * Investigate
      * Permissions problems (bug 840926) (Tomcat)
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * RDP should be disabled entirely (Chris)

     * Determine which show-desktop method works (two are in place now) (Chris)
     * Buildbot install script should use a different source for packages than scl1-production-puppet (dustin)
     * Install GVim using an Application rather than an MSI
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
 * any other issues discovered in testing

    
    

    
  
Virtual Audio Driver should be resolved, will need someone to verify who knows what they are looking for, but it appears to be installed.  Documentation in the Google Doc.

    
    

    
  
Win8 TS now joins the domain, goes to OU=windows,OU=machines,DC=releng,DC=AD,DC=Mozilla,DC=COM

Will figure out how to get it in proper ou's for location later.

    
    

    
  
 * Blocker
   * Change
     * Buildbot scheduled task does not run w/ highest privileges (+ documentation) (Chris))
     * does not install "Mozilla Maintenance Service" (Chris)
     * does not handle secondary graphics card - (Q and Chris)
     * Synchronous GPO Updates (Q and Chris)
     * Synchronous Startup / Login Scripts (Q and Chris)
     * Replace the current pywin install (with an undocumented MSI) with an Application (Q and Chris)
     * Set up Mozharness virtualenv at c:\slave\test\build\venv (not c:\talos-slave) (Tomcat)
   * Verify
     * Windows Activation should "just work" when hosts are added to the domain
     * Verify that C:\mozilla-build\python is adequate (armen)
    * Investigate
      * Permissions problems (bug 840926) (Tomcat)
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * RDP should be disabled entirely (Chris)
     * Buildbot install script should use a different source for packages than scl1-production-puppet (dustin)
     * Install GVim using an Application rather than an MSI
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
 * any other issues discovered in testing

    
    

    
  
Buildbot scheduled task does not run w/ highest privileges (+ documentation) (Chris))

Is that Talos?  That's the only scheduled task item i see being created in the TS

    
    

    
  
Yes, that's correct.  It's under "Buildbot Startup Process Configured" in the google doc.

    
    

    
  
Per Armen:
 * We *do* need Python-2.7.3 installed at C:\mozilla-build\python27, and PyWin installed in that
 * We can *remove* c:\mozilla-build\python
   - this will require a tiny modification to the Buildbot install process to use Python-2.7.3

I'll update the google doc accordingly, and since we have a meeting in an hour, update this bug and the various etherpads that have copies of it.

    
    

    
  
Updated.  I've shortened the text on a few of these - hopefully the identity of each item is still clear.

Windows activation needs purchase of a new key, which could take a while, so I moved it to the non-blocker category.  Of course, we'll need to move quickly on that, but it can easily be fixed for already-imaged hosts once the new key has been purchased.

 * Blocker
   * Change
     * Buildbot scheduled task does not run w/ highest privileges (+ documentation) (Chris)
     * Mozilla Maintenance Service installed (Chris)
     * Secondary graphics card automatically set up and used (Q and Chris)
     * Synchronous GPO Updates (Q and Chris)
     * Synchronous Startup / Login Scripts (Q and Chris)
     * Set up Mozharness virtualenv at c:\slave\test\build\venv (not c:\talos-slave) (Tomcat)
     * Delete c:\mozilla-build\python after installing MozillaBuild (documented, but not done - Chris)
     * Install Python-2.7.3 (documented, but not done - Chris)
     * Install PyWin-218 in Python-2.7.3 (documented, but not done - Chris)
     * Adjust Buildbot install process to use Python-2.7.3 (documented, but not done - Chris)
     * Adjust PATH to include Python-2.7.3 (documented, but not done - Chris)
    * Investigate
     * Permissions problems (bug 840926) (Tomcat)
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * Windows Activation - need a new KMS key covering Windows 8 (Q)
     * RDP should be disabled entirely (Chris)
     * Buildbot install script should use a different source for packages than scl1-production-puppet (dustin)
     * Install GVim using an Application rather than an MSI
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
 * any other issues discovered in testing

    
    

    
  
     * Buildbot scheduled task does not run w/ highest privileges 

Configured to run as Administrator

     * does not install "Mozilla Maintenance Service" (Chris)

Created MDT application to handle install, scheduled task creation and certificate import

    
    

    
  
-GPO/startup stuff i'll just leave for Q to handle with AD
-Graphics configuration, i know Q has manual process for it, my recommendation is to use either a VBS with sendkeys, or use AutoIt to capture the process, and then we can see if calling that during the TS works

    
    

    
  
Armen took care of

     * Set up Mozharness virtualenv at c:\slave\test\build\venv (not c:\talos-slave) (Tomcat)

    
    

    
  
     * Delete c:\mozilla-build\python after installing MozillaBuild (documented, but not done - Chris)
     * Install Python-2.7.3 (documented, but not done - Chris)
     * Install PyWin-218 in Python-2.7.3 (documented, but not done - Chris)
     * Adjust Buildbot install process to use Python-2.7.3 (documented, but not done - Chris)
     * Adjust PATH to include Python-2.7.3 (documented, but not done - Chris)
    * Investigate

Should all be addressed, will run through test to verify. All tasks are updated in google doc with how they were addressed.

    
    

    
  
Once verified on the new build, this should be the new list:

* Blocker
   * Change
     * Secondary graphics card automatically set up and used (Q and Chris)
     * Synchronous GPO Updates (Q and Chris)
     * Synchronous Startup / Login Scripts (Q and Chris)
     * Set up Mozharness virtualenv at c:\slave\test\build\venv (not c:\talos-slave) (Tomcat)
    * Investigate
     * Permissions problems (bug 840926) (Tomcat)
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * Windows Activation - need a new KMS key covering Windows 8 (Q)
     * RDP should be disabled entirely (Chris)
     * Buildbot install script should use a different source for packages than scl1-production-puppet (dustin)
     * Install GVim using an Application rather than an MSI
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker
 * any other issues discovered in testing

    
    

    
  
Per bug 841720, we need to quickly fix up the Buildbot install problems and find/fix the MMS scheduled task, then re-image two machines and start round III of testing.  Everything else is looking good!

    
    

    
  
Tweaked the order for the buildbot stuff and will verify in the image.

Plus i just wanted comment 100 :)

    
    

    
  
https://bugzilla.mozilla.org/show_bug.cgi?id=836999#c30

update to this bug, seems we were able to workaround Bug 840926 (the permission/rights problem) - patch there now inside.

01:26:40     INFO - Return code: 0
01:26:40     INFO - TinderboxPrint: reftest-crashtest<br/>2225/0/15
01:26:40     INFO - # TBPL SUCCESS #
01:26:40     INFO - The reftest suite: crashtest ran with return status: SUCCESS
01:26:40     INFO - Copying logs to upload dir...
01:26:40     INFO - mkdir: C:\slave\test\build\upload\logs

\o/

    
    

    
  
Blocker update 
     * Synchronous GPO Updates (Q and Chris)
     * Synchronous Startup / Login Scripts (Q and Chris)

GPO is processed synchronously by default. A new GPO called "Synchronous_processing" sets all logon activity to synchronous has been created and is applied to all non server and non DC machines in the releng domain.

    
    

    
  
So my understanding of the status is

 * Blocker
   * Change
     * Buildbot scheduled task does not run w/ highest privileges (+ documentation) (Chris)
     * Secondary graphics card automatically set up and used (Q and Chris)
 * Non-Blocker
   * Documentation
     * See and address all comments in the google doc
   * Change
     * RDP should be disabled entirely (Chris)
     * Buildbot install script should use a different source for packages than scl1-production-puppet (dustin)
     * Install GVim using an Application rather than an MSI
     * not pingable (needs firewall rule) (Chris) - not a blocker?
     * no SSH client, maybe copSSH?  (Armen to work with Q to specify, install, test)
     * patched twisted? (still needs detail - Armen, Dustin) - not a blocker, but a technical improvement that we need         
        * Deploy _dumbwin32proc.py that allows buildbot to kill jobs - armenzg to figure it out
     * GPO/puppet configuration management - not a blocker

    
  
Depends on: 844130

    
    

    
  
.Net Framework 3.5 added to deployment Task Sequence, verified installation

    
    

    
  
cltbld start-up scheduled tasks no handled in GPO in the releng domain and linked to the "tester" ous. No longer needed in the install Task Sequences.

    
    

    
  
I spoke with armen this morning, and he says that from his perspective, all of the testing works and he hopes to have this first batch into production by tomorrow.  We are still having some issues with automation of the initial screen creation for the 610, but we're not going to block releng on that.  We're going to keep one machine so that we can continue to test the automation of the graphics card, but we'll be rolling out the rest of these machines (doing that one step by hand) and handing them over to releng this week.
Blocks: 846550

    
    

    
  
I'm breaking this out into sub-bugs, mostly because this one takes forever to scroll through.  I'm not marking these as blockers since, well, they're not blockers.

* Secondary graphics card automatically set up and used - bug 847408
* See and address all comments in the google doc - bug 847407
* RDP should be disabled entirely - DONE
* Buildbot install script should use a different source for packages than scl1-production-puppet - bug 847400
* Install GVim using an Application rather than an MSI - bug 847402
* not pingable (needs firewall rule) - DONE
* no SSH client - bug 792456
* patched twisted? - bug 666019
* GPO/puppet configuration management - ongoing process, e.g., bug 798657

Given all that, I'm going to ambitiously close this bug, but please re-open if there's more to do here.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
No longer depends on: 792456
Resolution: --- → FIXED

    
    

    
  
This is actually the tracker bug, so shouldn't be closed until the automation to roll out a production system is finished.  Right now that includes finishing the graphics automation and getting ssh installed.
Status: RESOLVED → REOPENED
Depends on: 847408, 792456
Resolution: FIXED → ---

    
  
Blocks: win8-testing
No longer blocks: metro-testing

    
    

    
  
The graphics process is set to be automated with a send keys script. It will test to see if there is only one screen at logon then add the second monitor. If both "monitors" exist and the cltbld user is logged in it will double check resolution and make sure the primary monitor is running on the nvidia GPU. I am still doing final testing in a small scope with a GPO today. Should be rolled out this week ( hopefully by tomorrow)

    
    

    
  
1) Display automatically added via a logon scheduled task called 1a_config_monitor that calls C:\Monitor_config\fakemon.vbs that is deployed via gpo.
2)    Task adds display if it not detected then changes resolution via configmymonitor.exe which is a custom application written in C#. (This will eventually replace fakemon.vbs)
3)    Notes:
        * This allows the inband card to remain active
        * During the time that the nvidia addon card is in use the console redirect doesn’t work but VNC does
        * The video will always revert to the onboard videocard during boot
Status: REOPENED → ASSIGNED
Depends on: 849827

    
    

    
  
Can we do this as well? Sooner or later we will be asked to update the graphic card's drivers. Might as well iron it out and be ready for when we are asked to do it again.

(In reply to Jim Mathies [:jimm] from comment #2)
> There is a newer video driver for these machines, have we considered
> updating to the latest?
> 
> http://www.geforce.com/drivers/results/57096

    
    

    
  
No moving the goalposts!  That should be a new bug.  We're not keeping this one open for five years until we throw out the hardware :)

    
    

    
  
My suggestion originated from my wondering if a newer driver might fix all the svg ref test failures we have in bug 848936. No idea if it would, currently waiting on some feedback in that bug from Bas.
Assignee: cnackers → q

    
    

    
  
To facilitate the driver updates, that portion is built into the deployment process and not the core reference image.  So if there is a newer version, we could add it to the deployment TS and the next machine deployed would reflect that driver.

My concerns would be:
1) if any changes impact the monitor configuration Q developed
2) anything else that might be impacted

Generally speaking, I would recommend the updated driver is tested elsewhere to make sure it doesn't cause any additional issues before we change the core process to use that driver.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: