827497 - Modifications to the W64 task sequence

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

* install the 8.0 SDK along with a patch for bug 774910

https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=29327649 suggests the following be done manually; they should be automatic:

* set the local administrator password during the task sequence (without including it in cleartext!)
* make the server-management app not appear on startup
* fix the drive mapping (second drive at e:, CD-ROM elsewhere)
* set up autologin for the cltbld user; if possible, don't include the user's cleartext password here

Comment 1

[Chris Nackers [:cnackers]]

COMPLETED -  install the 8.0 SDK along with a patch for bug 774910
COMPLETED -  make the server-management app not appear on startup
COMPLETED - fix the drive mapping (second drive at e:, CD-ROM elsewhere), this is dependent upon each hardware configuration
COMPLETED - set up autologin for the cltbld user; if possible, don't include the user's cleartext password here

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Great - I'll try reimaging the five machines we have waiting for this, and hand them to releng to have a look at.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Bad news :(

This is failing to run the "Upgrade Buildbot" step.  Running the script manually permission denied errors.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

So from what I see, the things to fix are:

1. Convert the batch file to an MDT application for better control
2. Don't try to run as an administrator (not necessary)
3. Move the BinScope MSI install into a separate TS step
4. Change the 'md' at the end to use E:
5. In a new step, copy http://hg.mozilla.org/build/puppet/file/tip/modules/buildslave/files/runslave.py to C:\ (make a local copy of the script in the TS, rather than fetching it from that URL).

Hopefully that should fix the problem.

Comment 5

[Chris Nackers [:cnackers]]

Moved the BuildBot Upgrade into a MDT Application.

Removed all wget calls from batch and instead we copy down an updated source at start then run the commands locally. This removes the external dependencies for the imaging process.  

MD is still set to use D:, because we don't shuffle the drives around until after MDT completes (After-action script)

Runslave.py is copied down as part of the MDT application.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I'm running this (from the production deployment share) on w64-ix-slave21 now.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

No luck: bug 820992 comment 6:

> http://dev-master01.build.scl1.mozilla.com:8052/builders/WINNT%205.
> 2%20mozilla-central%20xulrunner/builds/1/steps/clean_old_builds/logs/stdio
> 
> Traceback (most recent call last):
>   File "tools/buildfarm/maintenance/purge_builds.py", line 31, in <module>
>     import win32file
> ImportError: No module named win32file
> 
> while running
> 
> 'python' 'tools/buildfarm/maintenance/purge_builds.py' '-s' '12' '-n' 'info'
> '-n' 'rel-*' '..' '/mock/users/cltbld/home/cltbld/build'
> 
> with a PATH of
> 
> PATH=C:\mozilla-build\nsis-2.46u;C:\mozilla-build\python27;C:\mozilla-
> build\buildbotve\scripts;C:\mozilla-build\msys\local\bin;c:\mozilla-
> build\wget;c:\mozilla-build\7zip;c:\mozilla-build\blat261\full;c:\mozilla-
> build\python;c:\mozilla-build\svn-win32-1.6.3\bin;c:\mozilla-build\upx203w;c:
> \mozilla-build\emacs-22.3\bin;c:\mozilla-build\info-zip;c:\mozilla-
> build\nsis-2.22;c:\mozilla-build\nsis-2.33u;c:\mozilla-build\nsis-2.46u;c:
> \mozilla-build\wix-351728;c:\mozilla-build\hg;c:\mozilla-
> build\python\Scripts;c:\mozilla-build\kdiff3;c:\mozilla-build\yasm;.;C:
> \mozilla-build\msys\local\bin;C:\mozilla-build\msys\mingw\bin;C:\mozilla-
> build\msys\bin;c:\Program Files (x86)\Microsoft Visual Studio
> 9.0\VC\BIN\amd64;c:\Windows\Microsoft.NET\Framework64\v3.5;c:
> \Windows\Microsoft.NET\Framework64\v3.5\Microsoft .NET Framework 3.5
> (Pre-Release
> Version);c:\Windows\Microsoft.NET\Framework64\v2.0.50727;c:\Program Files
> (x86)\Microsoft Visual Studio
> 9.0\VC\VCPackages;c:\Tools\msvs9\Common7\IDE;c:\Tools\msvs9\Common7\Tools;c:
> \Tools\msvs9\Common7\Tools\bin;c:\Tools\sdks\v7.0\bin\x64;c:\Tools\sdks\v7.
> 0\bin\win64\x64;c:\Tools\sdks\v7.0\bin;c:\windows\System32;c:\windows;c:
> \windows\System32\Wbem;c:\mozilla-build\moztools-x64\bin;c:\mozilla-
> build\vim\vim72

> Do you know where win32file is expected to be installed?  As in, which
> Python?  Py27 is in that path before buildbotve, so it's being run instead
> of the buildbotve python.
> 
> I don't think we have any steps to install win32file.  I don't see it in
> https://wiki.mozilla.org/ReferencePlatforms/Win64

> I'm able to import win32file from c:\mozilla-build\python27\python.exe, and
> I see a win32file.pyd in c:\mozilla-build\python27\Lib\site-packages on
> w64-ix-slave22.
> 
> I'm unfamiliar with what it is or why we need it; I'll ask around.

> Evidently it's from pywin32, which is on that refplatform page:
> https://wiki.mozilla.org/ReferencePlatforms/Win64#Python_2.7.3

That's new stuff from bug 602908, I think.  The MSI install was in buildbot_update.bat, so that got done, but the second part was not.

This means adding the equivalent of

c:\mozilla-build\msys\bin\bash --login -i
/c/mozilla-build/wget/wget.exe http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/python/install_pywin32_python27.sh
./install_pywin32_python27.sh

to the install process.  The content of that shell script is

----
MOZILLABUILD=/c/mozilla-build
if [ -d /d/mozilla-build ]; then
  MOZILLABUILD=/d/mozilla-build
fi
rm -rf pywin32-temp
mkdir pywin32-temp
cd pywin32-temp
${MOZILLABUILD}/wget/wget http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/python/pywin32-218.win32-py2.7.exe
${MOZILLABUILD}/info-zip/unzip pywin32-218.win32-py2.7.exe
cp -r PLATLIB/* ${MOZILLABUILD}/python27/lib/site-packages
cp -r SCRIPTS/* ${MOZILLABUILD}/python27/lib/site-packages
pushd ${MOZILLABUILD}/python27/lib/site-packages
${MOZILLABUILD}/python27/python pywin32_postinstall.py -quiet -install
popd
cd ..
rm -rf pywin32-temp
----

This is an abomination, I know..

Basically that's getting pywin32-218.win32-py2.7.exe (which we'll get in \\releng\data\apps\tmp\bug827497 for you, Chris), unzipping it, and copying the contents of PLATLIB and SCRIPTS into c:\mozillabuild\python27\lib\site-packages.  Then it changes directory to c:\mozillabuild\python27\lib\site-packages and runs pywin32_postinstall.py with c:\mozilla-build\python27\python.exe.  I'm assuming the directory-change is critical there.

So: Chris, can you replicate that?  You can expense the anti-nausea pills.

Aki/Coop: are there other surprises from around the time this was implemented or later, that we can head off before we re-image and hand back to you again?

Comment 8

[Chris Nackers [:cnackers]]

Updated the build to account for this app:

msiexec /i %mozillabuild%\python-2.7.3.msi /qb /norestart  /log c:\logs\python27_install.log TARGETDIR=c:\mozilla-build\python27 ALLUSERS=1

SET python27=%mozillabuild%\python27

%python27%\python.exe %python27%\lib\site-packages\pywin32_postinstall.py -quiet -install

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Indeed, I can now import 'win32file' using c:\mozilla-build\python27\python.exe.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: firewall error

So, on one that ran a job already (w64-ix-slave11), this has a firewall warning.

Coop, let's talk on Tuesday about this Python-2.7.3 project.  I'm not sure where the project became de-synchronized with the MDT work Matt was doing, but we need to get it put back together in a sane fashion, rather than chasing our tail like this.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Chris and I don't see any existing firewall configuration for Python on other, known-good w64 builders.  Is this a known problem?

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Kim's verifying right now that the hosts did OK in staging over the weekend.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

This failed repeatedly on Aki's master over the weekend:

e:\builds\moz2_slave\m-cen-w64-ntly\build\toolkit\mozapps\installer\packager.mk:884:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py make-package-internal' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\config\rules.mk:582:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py libs' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\browser\build.mk:45:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py -C browser/installer' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\client.mk:205:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py -C obj-firefox package MOZ_PGO_INSTRUMENTED=1 MOZ_INTERNAL_SIGNING_FORMAT= MOZ_EXTERNAL_SIGNING_FORMAT=' failed, return code 2
e:\builds\moz2_slave\m-cen-w64-ntly\build\client.mk:160:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64-ntly/build/build/pymake/pymake/../make.py -f e:/builds/moz2_slave/m-cen-w64-ntly/build/client.mk profiledbuild' failed, return code 2

no data yet on what that means..

Comment 14

[Amy Rich [:arr] [:arich]]

coop: The things that we tried yesterday seemed to have failed as well.  We're blocked on this until we can get someone from releng who can help us define what this should look like and how it should work.

Comment 15

[Chris Cooper [:coop] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #14)
> coop: The things that we tried yesterday seemed to have failed as well. 
> We're blocked on this until we can get someone from releng who can help us
> define what this should look like and how it should work.

Is the failure the same as comment #13? 

I would note that the timing of the failures on Aki's test master coincides with bug 832992. Have we run another test series to rule that out? 

Has anyone logged into the machine after/during the build to check for errant dialog boxes?

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Those are all questions I can't answer (except the errant dialog boxes - see comment 10).  Kim?

Comment 17

[Kim Moir [:kmoir] ET]

I setup my dev-master to run builds on these machines yesterday but they are still red so I'm still investigating :-)  Will update when I know more.

Comment 18

[Kim Moir [:kmoir] ET]

I seem to have the same errors on these machines as they builds that are currently running in production

For instance, a WINNT 6.1 x86-64 mozilla-central build is failing make_pkg

e:\builds\moz2_slave\m-cen-w64\build\toolkit\mozapps\installer\packager.mk:588:0: command 'e:/builds/moz2_slave/m-cen-w64/build/obj-firefox/_virtualenv/Scripts/python.exe e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=firefox -DPREF_DIR=defaults/pref -D_MSC_VER=1600 -DJAREXT= -DMOZ_ANGLE_RENDERER=1 -DMOZ_D3DX9_DLL=d3dx9_43.dll -DMOZ_D3DCOMPILER_DLL=D3DCompiler_43.dll -DMOZ_CHILD_PROCESS_NAME=plugin-container.exe -DMOZ_MSVC_REDIST=1600 -DMOZ_SHARED_MOZGLUE=1 -DMOZ_JSDEBUGGER -DNECKO_WIFI -DDLL_PREFIX= -DDLL_SUFFIX=.dll -DBIN_SUFFIX=.exe -DBINPATH=bin \
	--format omni \
	--removals e:/builds/moz2_slave/m-cen-w64/build/browser/installer/removed-files.in \
	 \
	 \
	--jarlogs e:/builds/moz2_slave/m-cen-w64/build/obj-firefox/browser/installer/../../jarlog//en-US \
	--optimizejars \
	 \
	package-manifest ../../dist ../../dist/firefox' failed, return code 1
e:\builds\moz2_slave\m-cen-w64\build\toolkit\mozapps\installer\packager.mk:614:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64/build/build/pymake/pymake/../make.py make-package-internal' failed, return code 2
Traceback (most recent call last):
  File "e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py", line 362, in <module>
    main()
  File "e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py", line 354, in main
    args.source, gre_path, base)
  File "e:/builds/moz2_slave/m-cen-w64/build/toolkit/mozapps/installer/packager.py", line 142, in precompile_cache
    errors.fatal('Error while running startup cache precompilation')
  File "e:\builds\moz2_slave\m-cen-w64\build\python\mozbuild\mozpack\errors.py", line 101, in fatal
    self._handle(self.FATAL, msg)
  File "e:\builds\moz2_slave\m-cen-w64\build\python\mozbuild\mozpack\errors.py", line 96, in _handle
    raise ErrorMessage(msg)
mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
e:\builds\moz2_slave\m-cen-w64\build\config\rules.mk:582:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64/build/build/pymake/pymake/../make.py libs' failed, return code 2
e:\builds\moz2_slave\m-cen-w64\build\browser\build.mk:45:0: command 'C:/mozilla-build/python27/python.exe e:/builds/moz2_slave/m-cen-w64/build/build/pymake/pymake/../make.py -C browser/installer' failed, return code 2
program finished with exit code 2

but the same error is occurring on builds running on non-staging slaves.

I don't see any errant dialog boxes on the machine.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

OK, let's call it done!  Do you want to toss those five (10,11,21,42,43, iirc) into production?  Please open a new bug if you do find any further problems, or if the problem you've identified needs to be solved with changes to the image.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

(sorry, that's 10,20,21,42,43)

Comment 21

[Kim Moir [:kmoir] ET]

I didn't have time to run the other tests I wanted to run today because of other buildduty tasks, I'll look at this again on MOnday.

Comment 22

[Kim Moir [:kmoir] ET]

Additional tests were successful, win64-ix-slave11,20,21,42 and 43 are back in production.