565402 - Install tool-chain on win64-ix-ref

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Armen [:armenzg]]

The initial goal is to get a machine that:
1) can generate Windows 64bit builds
2) process jobs from buildbot
3) other slaves can be cloned of and be part of a builders pool
4) let's have in mind possibly generating 32bit builds (Let's not scope creep)

To archive the first point I will follow this:
* Read https://developer.mozilla.org/En/Developer_Guide/Build_Instructions/Windows_Prerequisites
* Install:
    * MozillaBuild 1.4
    * Microsoft Visual C++ 2010 Express
    * Microsoft Windows SDK(s)
         * Is Windows7 SDK enough?

After I hand out the first 64 bit build to someone to QA points 2 and 3 should be tackled.

Other tools that will need to be installed:
    * Buildbot
    * NRPE daemon
    * OPSI
    * VNC
    * ssh

Check https://wiki.mozilla.org/ReferencePlatforms/Win32#Configuration_Details and see which configuration changes are still applicable for this new machine. So far I have seen these ones:
# disable automatic updates
# indexing
# registry changes
# environment vars
# any extra twistd patches?
# ssh keys (accept staging keys)
# screen resolution for unit tests
# java
# autoit
# opsi
    * timeouts
    * disable "shutdown event tracker"
    * OPSI hostkey generator
# buildbot tac generator
# cltbld autologin

Comment 1

[[e-mail] (:bhearsum)]

Note that some files, such as buildbot-tac.py, the buildbot.bat startup file, and possibly others will need to be updated to be made aware of the win64 hostnames.

Comment 2

[Armen [:armenzg]]

Attachment: first build log

I think that I have set up the requirements correctly (I have passed configure) but I am getting build errors.

c:/Users/Administrator/armenzg/mozilla-central/build/win32/vmwarerecordinghelper/vmwarerecordinghelper.cpp(66) : error C2143: syntax error : missing ';' before '}'

Here are my settings:
* MS 2008 SP2 (x64 bit)
* VS2010 Pro
* Win7 SDK
* MozillaBuild 1.4 (plus msvc10-x64.bat)

> . $topsrcdir/browser/config/mozconfig
> 
> mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../objdir
> ac_add_options --target=x86_64-pc-mingw32
> ac_add_options --host=x86_64-pc-mingw32
> ac_add_options --disable-ogg

Comment 3

[Armen [:armenzg]]

I modified the "build/Makefile.in" file to don't build "win32" and I got the build all the way through (I will file a bug).

Please find the builds in here:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/
and as one-time prof-of-concept in here:
http://people.mozilla.com/~armenzg/win64/

Comment 4

[Armen [:armenzg]]

Attachment: buildbot.bat startup script changes

I am also adding "win64-" just in case one day we have 64 bit machines that are not ix machines and this way we won't forget about making the change. Your call.

What I have locally has different paths since I currently have the C drive but I am assuming that I will use the same paths once we create the others drives.

Comment 5

[[Baboo]]

You might be interested in the fact that WinSDK 7.1 was already released.

Comment 6

[Armen [:armenzg]]

Attachment: start-buildbot.bat for a Windows 64 bit machine with VC2010

This is pretty much this file:
http://hg.mozilla.org/mozilla-build/raw-file/224a69b28acf/start-msvc10-x64.bat
where I have added at the end the following:
> :start
> "%MOZILLABUILD%\msys\bin\bash" --login -c /d/mozilla-build/start-buildbot.sh
> "%MOZILLABUILD%\msys\bin\sleep" 30
removed:
> %MOZILLABUILD%\msys\bin\bash" --login -i" 
and added the same type of logging that appears in this file:
http://hg.mozilla.org/build/opsi-package-sources/raw-file/a467e0ee51ff/buildbot-startup/CLIENT_DATA/start-buildbot.bat

bhearsum I will post a diff after this attachment.

Comment 7

[Armen [:armenzg]]

Attachment: diff between what I am running on win64-ix-ref and what is being used on win32 machines

Differences with this file:
http://hg.mozilla.org/build/opsi-package-sources/raw-file/a467e0ee51ff/buildbot-startup/CLIENT_DATA/start-buildbot.bat

bhearsum what are your thoughts on this file?
Maybe we can call it from buildbot.bat (the startup script) with a parameter to either start on VC10 mode or VC8 mode so it can be shared with win32 and win64 machines.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 448518 [details] [diff] [review]
buildbot.bat startup script changes

Seems fine

Comment 9

[Armen [:armenzg]]

Attachment: start-buildbot.bat for a Windows 64 bit machine with VC2010 (v1)

Now that I have C, D & E drives on the w64-ix-ref machine I can update this attachment by changing /c/mozilla-build to /d/mozilla-build.

Comment 10

[Armen [:armenzg]]

Yasm has been deployed to the slave https://wiki.mozilla.org/ReferencePlatforms/Win64#Add_yasm.exe_to_PATH

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 448589 [details] [diff] [review]
diff between what I am running on win64-ix-ref and what is being used on win32 machines

>--- /c/Users/Administrator/Desktop/start-buildbot.bat   2010-06-01 10:46:21 -0700
>+++ start-buildbot.bat  2010-06-01 12:18:04 -0700
>@@ -1,69 +1,91 @@
> @echo off
> 
>-SET MOZ_MSVCVERSION=8
>-
>-SET MOZILLABUILDDRIVE=%~d0%
>-SET MOZILLABUILDPATH=%~p0%
>-SET MOZILLABUILD=%MOZILLABUILDDRIVE%%MOZILLABUILDPATH%
>+SET MOZ_MSVCVERSION=9
>+SET MOZBUILDDIR=%~dp0
>+SET MOZILLABUILD=%MOZBUILDDIR%
> 
> set log="c:\tmp\buildbot-startup.log"
> 
>-echo "Mozilla tools directory: %MOZILLABUILD%"
>+echo "Mozilla tools directory: %MOZBUILDDIR%"
> 
> REM Get MSVC paths
> echo "%date% %time% - About to call guess-msvc.bat" >> %log%
>-call "%MOZILLABUILD%\guess-msvc.bat"
>+call "%MOZBUILDDIR%guess-msvc.bat"
> 
> REM Use the "new" moztools-static
>-set MOZ_TOOLS=%MOZILLABUILD%\moztools
>+set MOZ_TOOLS=%MOZBUILDDIR%moztools-x64
> 
> rem append moztools to PATH
> SET PATH=%PATH%;%MOZ_TOOLS%\bin
> 
>-if "%VC8DIR%"=="" (
>-    if "%VC8EXPRESSDIR%"=="" (
>-        echo "%date% %time% - MSVC++8 not found" >> %log%
>-        ECHO "Microsoft Visual C++ version 8 was not found. Exiting."
>+if "%VC10DIR%"=="" (
>+    if "%VC10EXPRESSDIR%"=="" (
>+   echo "%date% %time% - MSVC++10  not found" >> %log%
>+        ECHO "Microsoft Visual C++ version 10 (2010) was not found. Exiting."
>         pause
>         EXIT /B 1
>     )
> 
>     if "%SDKDIR%"=="" (
>         echo "%date% %time% - SDK not found" >> %log%
>         ECHO "Microsoft Platform SDK was not found. Exiting."
>         pause
>         EXIT /B 1
>     )
> 
>     rem Prepend MSVC paths
>-    echo "%date% %time% - About to call vcvars32.bat" >> %log%
>-    call "%VC8EXPRESSDIR%\Bin\vcvars32.bat"
>+    echo "%date% %time% - About to call vcvars64.bat" >> %log%
>+    call "%VC10EXPRESSDIR%\bin\amd64\vcvars64.bat"
> 
>-    SET USESDK=1
>     rem Don't set SDK paths in this block, because blocks are early-evaluated.
>-) else (
>-    rem Prepend MSVC paths
>-    echo "%date% %time% - Calling vcvars32.bat in VC8DIR" >> %log%
>-    call "%VC8DIR%\Bin\vcvars32.bat"
> 
>-    rem If the SDK is Win2k3SP2 or higher, we want to use it
>-    if %SDKVER% GEQ 5 (
>-      SET USESDK=1
>+    rem Fix problem with VC++Express Edition
>+    if "%SDKVER%" GEQ "6" (
>+        rem SDK Ver.6.0 (Windows Vista SDK) and newer
>+        rem do not contain ATL header files.
>+        rem We need to use the Platform SDK's ATL header files.
>+        SET USEPSDKATL=1
>+    )
>+    rem SDK ver.6.0 does not contain OleAcc.idl
>+    rem We need to use the Platform SDK's OleAcc.idl
>+    if "%SDKVER%" == "6" (
>+        if "%SDKMINORVER%" == "0" (
>+          SET USEPSDKIDL=1
>     )
> )
>+) else (
>+    rem Prepend MSVC paths
>+    rem The Win7 SDK (or newer) should automatically integrate itself into vcvars32.bat
>+    ECHO Using VC 2010 built-in SDK
>+    echo "%date% %time% - Calling vcvars64.bat in VC10DIR" >> %log%
>+    call "%VC10DIR%\bin\amd64\vcvars64.bat"
>+)
> 
>-if "%USESDK%"=="1" (
>+if "%VC10DIR%"=="" (
>     rem Prepend SDK paths - Don't use the SDK SetEnv.cmd because it pulls in
>     rem random VC paths which we don't want.
>     rem Add the atlthunk compat library to the end of our LIB
>-    set PATH=%SDKDIR%\bin;%PATH%
>-    set LIB=%SDKDIR%\lib;%LIB%;%MOZILLABUILD%\atlthunk_compat
>-    set INCLUDE=%SDKDIR%\include;%SDKDIR%\include\atl;%INCLUDE%
>+    set "PATH=%SDKDIR%\bin\x64;%SDKDIR%\bin;%PATH%"
>+    set "LIB=%SDKDIR%\lib\x64;%SDKDIR%\lib;%LIB%;%MOZBUILDDIR%atlthunk_compat"

Why do you have quotes around the whole thing here?  And do you need a ';' before atlthunk_compat, or is that intentional?

Other than that, looks ok to me.  I don't really know what I'm looking at though :)  Maybe poke ted for a review?

>+
>+    if "%USEPSDKATL%"=="1" (
>+        if "%USEPSDKIDL%"=="1" (
>+            set "INCLUDE=%SDKDIR%\include;%PSDKDIR%\include\atl;%PSDKDIR%\include;%INCLUDE%"
>+        ) else (
>+            set "INCLUDE=%SDKDIR%\include;%PSDKDIR%\include\atl;%INCLUDE%"
>+        )
>+    ) else (
>+        if "%USEPSDKIDL%"=="1" (
>+            set "INCLUDE=%SDKDIR%\include;%SDKDIR%\include\atl;%PSDKDIR%\include;%INCLUDE%"
>+        ) else (
>+            set "INCLUDE=%SDKDIR%\include;%SDKDIR%\include\atl;%INCLUDE%"
>+        )
>+    )
> )
> 
> cd "%USERPROFILE%"
> :start
> echo "%date% %time% - About to run start-buildbot.sh" >> %log%
> "%MOZILLABUILD%\msys\bin\bash" --login -c /d/mozilla-build/start-buildbot.sh
> echo "%date% %time% - start-buildbot.sh finished" >> %log%
> "%MOZILLABUILD%\msys\bin\sleep" 30

Comment 12

[Armen [:armenzg]]

Attachment: python25.reg

Comment 13

[Armen [:armenzg]]

Attachment: buildbot-tac.py changes for win64

Comment 14

[Armen [:armenzg]]

Attachment: start-buildbot.bat based off start-msvc9-x64.bat

Comment 15

[Armen [:armenzg]]

Attachment: start-buildbot.sh for MozillaBuild1.5.1

The only difference is this:
- start_cmd="/d/mozilla-build/python/scripts/buildbot start $slave_dir"
+ start_cmd="/d/mozilla-build/python25/scripts/buildbot start $slave_dir"

Comment 16

[Armen [:armenzg]]

Attachment: python26.reg

MozillaBuild1.5.1 comes with python2.6 instead of 2.5 so I had to register 2.6 instead.

Comment 17

[Armen [:armenzg]]

Attachment: buildbot.bat for win64 without passwords

Comment 18

[Armen [:armenzg]]

Status update
=============
* I wasn't able to install OPSI after several days trying (see bug 596337)
* After we got OPSI working we would need to install nagios and few easier things
* Right now this work has very low priority as it does not block FF4 and there is just so much in my plate that can improve the release of FF4
* Before tackling this again I need to setup the Linux64 IX reference machine as it is a FF4 blocking platform

Comment 19

[Armen [:armenzg]]

Attachment: fix setting tools dir for Win64

This patch fixes setting the right toolsdir for Win64.

Comment 20

[Armen [:armenzg]]

Attachment: Start up buildbot.bat for win64 without passwords (v1)

Comment 21

[Armen [:armenzg]]

Attachment: start-buildbot.bat

Comment 22

[Armen [:armenzg]]

Attachment: start-buildbot.bat (take 2)

Comment 23

[Armen [:armenzg]]

We had to disable the Windows Error Reporting to avoid timing out on |make -k check|.
https://wiki.mozilla.org/ReferencePlatforms/Win64#Disable_Windows_Error_Reporting

Comment 24

[Armen [:armenzg]]

From my reading of:
http://technet.microsoft.com/en-us/library/cc786709%28WS.10%29.aspx

It seems that we don't need to do the 4GT tuning (aka /3GB switch) as done on the Win2k3 32-bit machines on bug 543034 [1].

The only thing that I noticed that might be needed is to set the IMAGE_FILE_LARGE_ADDRESS_AWARE flag [2].


[1]:
> 4-gigabyte tuning (4GT), also known as application memory tuning, or the /3GB
> switch, is a technology (only applicable to 32 bit systems) that alters the
> amount of virtual address space available to user mode applications. Enabling
> this technology reduces the overall size of the system virtual address space
> and therefore system resource maximums.

[2]:
> 2 GB with IMAGE_FILE_LARGE_ADDRESS_AWARE cleared (default)
> 4 GB with IMAGE_FILE_LARGE_ADDRESS_AWARE set

Comment 25

[Armen [:armenzg]]

Attachment: [buildbotcustom] add win64 to our custom libraries (take 2)

This addresses the issues you raised in https://bugzilla.mozilla.org/show_bug.cgi?id=567154#attachflag4

I am going to join the work of bug 567154 to this bug as it will make it harder to figure out what got fixed where.

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 520752 [details] [diff] [review]
[buildbotcustom] add win64 to our custom libraries (take 2)

You should add something to reallyShort() in misc.py, too.

Looks fine otherwise!

Comment 27

[Armen [:armenzg]]

Status update:
I have updated the Win64 reference platform documentation [1] and have requested machines to be cloned off it.

Next week I will hope to have them on staging and until then I will be landing the patches which I will ask for review as they get approved.

I will also be looking at the current state of the Win7x64 testing machines.

I am going to update some patches and change some bug dependencies to reflect reality.
Sorry in advance for bugmail produced from it!

[1] https://wiki.mozilla.org/ReferencePlatforms/Win64

Comment 28

[Armen [:armenzg]]

Comment on attachment 520752 [details] [diff] [review]
[buildbotcustom] add win64 to our custom libraries (take 2)

http://hg.mozilla.org/build/buildbotcustom/rev/18f466407438

Comment 29

[Armen [:armenzg]]

Comment on attachment 486067 [details] [diff] [review]
fix setting tools dir for Win64

Landed with attachment 520752 [details] [diff] [review].

Comment 30

[Armen [:armenzg]]

Attachment: [mozconfig] win64 mozconfigs v1

This got checked-in on bug 567154#c21 (pasting on this bug to keep everything in the same place):
http://hg.mozilla.org/build/buildbot-configs/rev/61f8b2f31012

Comment 31

[Armen [:armenzg]]

Nothing to be done until bug 645024 is fixed.

Comment 32

[Armen [:armenzg]]

(In reply to comment #31)
> Nothing to be done until bug 645024 is fixed.

As mentioned ^^

Comment 33

[Armen [:armenzg]]

Attachment: start-buildbot.bat (take 3) plus call /c/runslave.py

Comment 34

[Armen [:armenzg]]

Attachment: start-buildbot.bat (take 4) plus call /c/runslave.py

This patch is pretty much as attachment 519761 [details] which I know to be working but added at the end the chunk to call runslave.py.

Comment 35

[Armen [:armenzg]]

Comment on attachment 519478 [details]
Start up buildbot.bat for win64 without passwords (v1)

This attachment is obsolete by this:
http://hg.mozilla.org/build/opsi-package-sources/raw-file/2d51f32ff0f2/buildbot-startup/CLIENT_DATA/buildbot.bat

Comment 36

[Armen [:armenzg]]

Comment on attachment 473651 [details]
start-buildbot.sh for MozillaBuild1.5.1

This is obsoleted by slave-alloc support.

Comment 37

[Armen [:armenzg]]

Comment on attachment 473649 [details] [diff] [review]
buildbot-tac.py changes for win64

Obsoleted by slave-alloc support.

Comment 38

[Armen [:armenzg]]

Comment on attachment 448518 [details] [diff] [review]
buildbot.bat startup script changes

Obsoleted by slave-alloc support.

Comment 39

[Armen [:armenzg]]

Tools are setup and the machine is ready.

If we need to reopen or file following bugs then we will.

Comment 40

[Armen [:armenzg]]

Attachment: logon.reg that also removes AutoLogonCount

This is part of work in bug 645024.

Comment 41

[Armen [:armenzg]]

Attachment: runslave.py - this version creates E:\builds\moz2_slave if it doesn't exist

Landed as:
http://hg.mozilla.org/build/puppet-manifests/diff/1bbb72f7dcea/modules/buildslave/files/runslave.py