806799 - Switch custom elm builders over to vs2010 / 8.0 sdk

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Jim Mathies [:jimm]]

I'll sort out the path info and post a patch. We'll also need to make the header change detailed in bug 774910.

Comment 1

[Jim Mathies [:jimm]]

A couple other things we should test:

- turn pgo nightly builds back on
- flip to pymake to see if the build problems on elm have been addressed

Comment 2

[Jim Mathies [:jimm]]

Path info:

Elm's vs11 path info:
----------------------------------------------------------

INCLUDE=

/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/vc/include:
/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/vc/atlmfc/include:
/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/include/shared:
/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/include/um:
/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/include/winrt

LIBPATH=

/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/vc/lib:
/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/vc/atlmfc/lib:
/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/vc/vcpackages:
/c/Program\ Files\ \(x86\)/Microsoft\ SDKs/Windows/v8.0/ExtensionSDKs/Microsoft.VCLibs/11.0/References/CommonConfiguration/neutral:
/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/References/CommonConfiguration/Neutral

LIB=

/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/vc/lib:
/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/vc/atlmfc/lib:
/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/lib/win8/um/x86

PATH="

/c/Program Files (x86)/Microsoft Visual Studio 11.0/Common7/IDE:
/c/Program Files (x86)/Microsoft Visual Studio 11.0/VC/BIN:
/c/Program Files (x86)/Microsoft Visual Studio 11.0/Common7/Tools:
/c/Program Files (x86)/Microsoft Visual Studio 11.0/VC/vcpackages:
/c/Program Files (x86)/Windows Kits/8.0/bin/x86:
${PATH}"

WIN32_REDIST_DIR="/c/Program Files (x86)/Microsoft Visual Studio 11.0/VC/redist/x86/Microsoft.VC110.CRT"

WINDOWSSDKDIR="/c/Program Files (x86)/Windows Kits/8.0/"


Elm's vs10 path info:
----------------------------------------------------------

INCLUDE=

/c/tools/msvs10/vc/include:
/c/tools/msvs10/vc/atlmfc/include:
/c/tools/sdks/v7.0/include:
/c/tools/sdks/v7.0/include/atl:
/c/tools/sdks/dx10/include

LIBPATH=

/c/tools/msvs10/vc/lib:
/c/tools/msvs10/vc/atlmfc/lib

LIB=

/c/tools/msvs10/vc/lib:
/c/tools/msvs10/vc/atlmfc/lib:
/c/tools/sdks/v7.0/lib:
/c/tools/sdks/dx10/lib

PATH="

/c/tools/msvs10/Common7/IDE:
/c/tools/msvs10/VC/BIN:
/c/tools/msvs10/Common7/Tools:
/c/tools/msvs10/VC/VCPackages:
/c/mozilla-build/moztools:
/c/Tools/sdks/v7.0/bin:
${PATH}"

WIN32_REDIST_DIR=/c/tools/msvs10/VC/redist/x86/Microsoft.VC100.CRT

MOZ_TOOLS=C:/mozilla-build/moztools


Mixed vs10 / 8.0 sdk
----------------------------------------------------------

INCLUDE=

/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/include/shared:
/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/include/um:
/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/include/winrt:
/c/tools/msvs10/vc/include:
/c/tools/msvs10/vc/atlmfc/include

LIBPATH=

/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/Lib/win8/um/x86:
/c/tools/msvs10/vc/lib:
/c/tools/msvs10/vc/atlmfc/lib

LIB=

/c/Program\ Files\ \(x86\)/Windows\ Kits/8.0/Lib/win8/um/x86:
/c/tools/msvs10/vc/lib:
/c/tools/msvs10/vc/atlmfc/lib

PATH="

/c/tools/msvs10/Common7/IDE:
/c/tools/msvs10/VC/BIN:
/c/tools/msvs10/Common7/Tools:
/c/tools/msvs10/VC/VCPackages:
/c/mozilla-build/moztools:
/c/Tools/sdks/v7.0/bin:
${PATH}"

WIN32_REDIST_DIR="/c/Program Files (x86)/Microsoft Visual Studio 11.0/VC/redist/x86/Microsoft.VC110.CRT"

MOZ_TOOLS=C:/mozilla-build/moztools

Comment 3

[Jim Mathies [:jimm]]

Attachment: config patch v.1

Comment 4

[Jim Mathies [:jimm]]

Ok, the backport has landed.

Armen, can we pull one of our elm builders off line and run some test builds with this new setup? We'll also need to make the header change detailed in bug 774910 on each builder before this goes live on elm.

Comment 5

[Armen [:armenzg]]

I will start working on this between today and tomorrow.

Comment 6

[Jim Mathies [:jimm]]

Attachment: build script patch v.2

per the comments in bug 799188 I've swapped out the 7.0 sdk bin path with the 8.0 kit bin path.

Comment 8

[Armen [:armenzg]]

BTW, am I ready to test this on staging? Anything else to be landed?

We have 4 slaves on Elm:
* w64-ix-slave11
* w64-ix-slave20
* w64-ix-slave42
* w64-ix-slave43

Work to be done in here AFAIU:
* take the 4 w64 slaves offline and...
** install Windows 8 SDK [1]
** patch asyncinfo.h [2]
* test those slaves on staging
* put those slaves back to Elm
* determine if we're ready to deploy the changes to all w64 slaves

Does this make sense>

[1]
wget -O%USERPROFILE%\Dekstop http://dev-stage01.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/sdksetup.exe
sdksetup.exe /q /norestart
[2]
C:\Program Files (x86)\Windows Kits\8.0\Include\winrt\asyncinfo.h
line 67:
enum class AsyncStatus {
to
enum /*class*/ AsyncStatus {
[3]

Comment 9

[Armen [:armenzg]]

The 4 slaves I mentioned had VS2012 installed on to them.
Do I need to install the SDK or just adjust the file mentioned?

Comment 10

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #9)
> The 4 slaves I mentioned had VS2012 installed on to them.
> Do I need to install the SDK or just adjust the file mentioned?

That is a really good question, unfortunately I don't have the answer. I bet that the sdk install will succeed even if the sdk is already there, it just wouldn't do anything. So it's probably safe to do the install just in case.

Comment 11

[Jim Mathies [:jimm]]

> Work to be done in here AFAIU:
> * take the 4 w64 slaves offline and...
> ** install Windows 8 SDK [1]
> ** patch asyncinfo.h [2]
> * test those slaves on staging
> * put those slaves back to Elm
> * determine if we're ready to deploy the changes to all w64 slaves

Looks good to me. Will staging use the same path info I've posted here for elm so we can confirm this setup will work with elm?

Comment 12

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #6)
> Created attachment 677053 [details] [diff] [review]
> build script patch v.2
> 
> per the comments in bug 799188 I've swapped out the 7.0 sdk bin path with
> the 8.0 kit bin path.

We can remove the ac_add_options --disable-webrtc from this patch.

Comment 13

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #11)
> > Work to be done in here AFAIU:
> > * take the 4 w64 slaves offline and...
> > ** install Windows 8 SDK [1]
> > ** patch asyncinfo.h [2]
> > * test those slaves on staging
> > * put those slaves back to Elm
> > * determine if we're ready to deploy the changes to all w64 slaves
> 
> Looks good to me. Will staging use the same path info I've posted here for
> elm so we can confirm this setup will work with elm?

If you have landed your changes to Elm already then we should be ready to go.

If not, if you could push your changes to try I could then test on staging your try push.

Comment 14

[Armen [:armenzg]]

Switching to VS2010 and Windows 8 SDK:
http://hg.mozilla.org/projects/elm/rev/520f8f286861

Comment 15

[Jim Mathies [:jimm]]

I pulled both patches temporarily while I tack down something related to crt runtime.

Comment 16

[Jim Mathies [:jimm]]

New patch set landed:

https://hg.mozilla.org/projects/elm/rev/52bbfd4ea011

and I clobbered elm.

Comment 17

[Jim Mathies [:jimm]]

Win debug (w64-ix-slave18) is burning due to a lack of a c compiler, but it looks like this is expected. I guess these builds weren't being done by the custom elm builders. 

> We have 4 slaves on Elm:
> * w64-ix-slave11
> * w64-ix-slave20
> * w64-ix-slave42
> * w64-ix-slave43

Comment 18

[Armen [:armenzg]]

Correct. The debug builders are not locked to the set of 4 slaves.

Comment 19

[Jim Mathies [:jimm]]

Ok, Win opt reached the point of erroring out at the asyncinfo header file.

https://tbpl.mozilla.org/php/getParsedLog.php?id=16761876&tree=Elm&full=1#error1

Comment 20

[Armen [:armenzg]]

I updated line 67 on all those 4 slaves.
Should it have worked?

Would you prefer to back out the patches and loan you a machine?

Comment 21

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #20)
> I updated line 67 on all those 4 slaves.
> Should it have worked?
> 
> Would you prefer to back out the patches and loan you a machine?

I can say with pretty high confidence that w64-ix-slave20 didn't get updaed by the time the build got to the winrt code - 

c:\program files (x86)\windows kits\8.0\include\winrt\AsyncInfo.h(66) : error C2332: 'enum' : missing tag name

c:\program files (x86)\windows kits\8.0\include\winrt\AsyncInfo.h(66) : error C2236: unexpected 'class' 'ABI::Windows::Foundation::AsyncStatus'.

That comment above is in reference to the 'class' symbol that should be commented out - 

enum class AsyncStatus {

to

enum /*class*/ AsyncStatus {


Can you confirm w64-ix-slave20 was updated? If so, lets retrigger.

(If you want I can connect to the slave as well.)

Comment 22

[Armen [:armenzg]]

I checked the slave and it has the change in it.
I've re-triggered a couple of more jobs.

Comment 23

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #22)
> I checked the slave and it has the change in it.
> I've re-triggered a couple of more jobs.

ok, sounds good.

Comment 24

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #22)
> I checked the slave and it has the change in it.
> I've re-triggered a couple of more jobs.

w64-ix-slave20 came up red on the second build. Can we connect to that slave and confirm the header change?

Comment 25

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #24)
> (In reply to Armen Zambrano G. [:armenzg] from comment #22)
> > I checked the slave and it has the change in it.
> > I've re-triggered a couple of more jobs.
> 
> w64-ix-slave20 came up red on the second build. Can we connect to that slave
> and confirm the header change?

Actually is there any way you could grab that header and attach it here so we can take a look at it?

Comment 26

[Armen [:armenzg]]

Attachment: asyncinfo.h

This is odd.
When I open the file with vim I get this:
enum /*class*/ AsyncStatus {
but using cat shows this:
enum class AsyncStatus {

Comment 27

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #26)
> Created attachment 678464 [details]
> asyncinfo.h
> 
> This is odd.
> When I open the file with vim I get this:
> enum /*class*/ AsyncStatus {
> but using cat shows this:
> enum class AsyncStatus {

The files in program files don't have write access for admins or users by default so your editor might not have the ability to edit them.

You can copy asyncinfo.h out, edit someplace and drop it back in. If you're using rdp, you should get a uac prompt on the copy back.

I'm not sure if this is something you can do with a terminal.

Comment 28

[Jim Mathies [:jimm]]

Once we get past the asyncinfo.h header hurdle, there are few things we'd like to get done here:

1) back out bug 787563 and trigger to see if pymake problems on elm have been cleared up.
2) re-enable PGO for nightly builds and trigger a PGO build to test.
3) switch Win debug over to custom elm builders.

Comment 29

[Jim Mathies [:jimm]]

Attachment: current config changes on elm

Comment 30

[Armen [:armenzg]]

Attachment: config changes

This patch adds:
* enable pymake
* enable pgo
* win32-debug builds done by custom builders

I will fix the header issue and test this patch on staging.

Comment 31

[Armen [:armenzg]]

Attachment: asyncinfo.h (new)

Comment 32

[Armen [:armenzg]]

Attachment: asyncinfo.h (old)

Comment 33

[Armen [:armenzg]]

Attachment: asyncinfo.h (new)

Comment 34

[Armen [:armenzg]]

This seems to do the trick.
I have updated the builders and triggered jobs for them.

# ssh as Administrator - removing/copying operations takes a bit of time
C:\mozilla-build\wget\wget.exe -Oasyncinfo.h https://bugzilla.mozilla.org/attachment.cgi?id=678752
rm "C:\Program Files (x86)\Windows Kits\8.0\Include\winrt\asyncinfo.h"
cp asyncinfo.h "C:\Program Files (x86)\Windows Kits\8.0\Include\winrt\asyncinfo.h"

Before:
07/25/2012  12:31 PM             7,161 asyncinfo.h
               1 File(s)          7,161 bytes
               0 Dir(s)  36,594,774,016 bytes free
After:
11/06/2012  07:45 AM             6,931 asyncinfo.h
               1 File(s)          6,931 bytes
               0 Dir(s)  36,597,796,864 bytes free

Comment 35

[Armen [:armenzg]]

It seems that the 4 slaves + w64-ix-slave79 are capable of compiling WINNT 5.2 builds after the change in comment 34.

w64-ix-slave79 (on staging) tried to run a nightly build with PGO but it seems that is giving us trouble:

'python' 'e:/builds/moz2_slave/elm-w32-ntly/build/build/pymake/make.py' '-f' 'client.mk' 'build' 'MOZ_BUILD_DATE=20121106085120' 'MOZ_PGO=1'
...
make.py[4]: Leaving directory 'e:\builds\moz2_slave\elm-w32-ntly\build\obj-firefox\browser\installer'
make.py[4]: Entering directory 'e:\builds\moz2_slave\elm-w32-ntly\build\obj-firefox\browser\installer'
Packaging JavaScript Shell...
e:\builds\moz2_slave\elm-w32-ntly\build\toolkit\mozapps\installer\packager.mk:890:0$ rm -f ../../dist/jsshell-win32.zip
e:\builds\moz2_slave\elm-w32-ntly\build\toolkit\mozapps\installer\packager.mk:891:0$ zip -9j ../../dist/jsshell-win32.zip ../../dist/bin/js.exe ../../dist/bin/mozglue.dll  ../../dist/bin/nspr4.dll ../../dist/bin/msvcr100.dll
  adding: js.exe (172 bytes security) (deflated 61%)
  adding: mozglue.dll (172 bytes security) (deflated 49%)
  adding: nspr4.dll (172 bytes security) (deflated 54%)
  adding: msvcr100.dll (172 bytes security) (deflated 46%)
e:\builds\moz2_slave\elm-w32-ntly\build\toolkit\mozapps\installer\packager.mk:755:0$ e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/_virtualenv/Scripts/python.exe e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=firefox -DPREF_DIR=defaults/preferences -D_MSC_VER=1600 -DJAREXT= -DMOZ_ANGLE_RENDERER=1 -DMOZ_D3DX9_DLL=d3dx9_43.dll -DMOZ_D3DCOMPILER_DLL=D3DCompiler_43.dll -DMOZ_CHILD_PROCESS_NAME=plugin-container.exe -DMOZ_MSVC_REDIST=1600 -DMOZ_SHARED_MOZGLUE=1 -DMOZ_JSDEBUGGER -DNECKO_WIFI -DDLL_PREFIX= -DDLL_SUFFIX=.dll -DBIN_SUFFIX=.exe -DBINPATH=bin --format omni --removals e:/builds/moz2_slave/elm-w32-ntly/build/browser/installer/removed-files.in e:/builds/moz2_slave/elm-w32-ntly/build/browser/installer/package.manifest ../../dist/bin ../../dist/firefox
e:\builds\moz2_slave\elm-w32-ntly\build\toolkit\mozapps\installer\packager.mk:755:0: command 'e:/builds/moz2_slave/elm-w32-ntly/build/obj-firefox/_virtualenv/Scripts/python.exe e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=firefox -DPREF_DIR=defaults/preferences -D_MSC_VER=1600 -DJAREXT= -DMOZ_ANGLE_RENDERER=1 -DMOZ_D3DX9_DLL=d3dx9_43.dll -DMOZ_D3DCOMPILER_DLL=D3DCompiler_43.dll -DMOZ_CHILD_PROCESS_NAME=plugin-container.exe -DMOZ_MSVC_REDIST=1600 -DMOZ_SHARED_MOZGLUE=1 -DMOZ_JSDEBUGGER -DNECKO_WIFI -DDLL_PREFIX= -DDLL_SUFFIX=.dll -DBIN_SUFFIX=.exe -DBINPATH=bin --format omni --removals e:/builds/moz2_slave/elm-w32-ntly/build/browser/installer/removed-files.in e:/builds/moz2_slave/elm-w32-ntly/build/browser/installer/package.manifest ../../dist/bin ../../dist/firefox' failed, return code 1
e:\builds\moz2_slave\elm-w32-ntly\build\config\rules.mk:556:0: command 'C:/mozilla-build/buildbotve/scripts/python.exe e:/builds/moz2_slave/elm-w32-ntly/build/build/pymake/pymake/../make.py libs' failed, return code 2
e:\builds\moz2_slave\elm-w32-ntly\build\browser\build.mk:47:0: command 'C:/mozilla-build/buildbotve/scripts/python.exe e:/builds/moz2_slave/elm-w32-ntly/build/build/pymake/pymake/../make.py -C browser/installer' failed, return code 2
e:\builds\moz2_slave\elm-w32-ntly\build\client.mk:197:0: command 'C:/mozilla-build/buildbotve/scripts/python.exe e:/builds/moz2_slave/elm-w32-ntly/build/build/pymake/pymake/../make.py -C obj-firefox package MOZ_PGO_INSTRUMENTED=1 MOZ_INTERNAL_SIGNING_FORMAT= MOZ_EXTERNAL_SIGNING_FORMAT=' failed, return code 2
e:\builds\moz2_slave\elm-w32-ntly\build\client.mk:152:0: command 'C:/mozilla-build/buildbotve/scripts/python.exe e:/builds/moz2_slave/elm-w32-ntly/build/build/pymake/pymake/../make.py -f e:/builds/moz2_slave/elm-w32-ntly/build/client.mk profiledbuild' failed, return code 2
res/spacetrace.css was not registered
Missing file(s): ..\..\dist\bin\CommandExecuteHandler.exe
Traceback (most recent call last):
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 600, in <module>
    main()
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 586, in main
    packager.finalize()
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 547, in finalize
    Packager.finalize(self)
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 357, in finalize
    self.copier.finalize()
  File "e:/builds/moz2_slave/elm-w32-ntly/build/toolkit/mozapps/installer/packager.py", line 180, in finalize
    raise Exception, 'Aborting due to errors'
Exception: Aborting due to errors
program finished with exit code 2
elapsedTime=3682.385000

Comment 36

[Armen [:armenzg]]

Here are the env variables:
  !EXITCODE=00000001
  ALLUSERSPROFILE=C:\ProgramData
  APPDATA=C:\Users\cltbld\AppData\Roaming
  APR_ICONV_PATH=c:/mozilla-build/svn-win32-1.6.3/iconv
  BINSCOPE=C:\Program Files (x86)\Microsoft\SDL BinScope\BinScope.exe
  COMMONPROGRAMFILES=C:\Program Files (x86)\Common Files
  COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
  COMMONPROGRAMW6432=C:\Program Files\Common Files
  COMPUTERNAME=W64-IX-SLAVE79
  COMSPEC=C:\windows\system32\cmd.exe
  CVS_RSH=ssh
  DXSDK_DIR=C:\Tools\sdks\dx10\
  EDITOR=emacs.exe --no-window-system
  FP_NO_HOST_CHECK=NO
  FRAMEWORK35VERSION=v3.5
  FRAMEWORKDIR=C:\Windows\Microsoft.NET\Framework64
  FRAMEWORKVERSION=v2.0.50727
  HG_SHARE_BASE_DIR=e:/builds/hg-shared
  HOME=c:/Users/cltbld
  HOMEDRIVE=C:
  HOMEPATH=\
  HOSTTYPE=i686
  INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE;C:\Tools\sdks\v7.0\include;;c:\tools\sdks\dx10\include
  INPUTRC=C:/mozilla-build/msys/etc/inputrc
  IS_NIGHTLY=yes
  LIB=C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\ATLMFC\LIB\amd64;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\LIB\amd64;C:\Tools\sdks\v7.0\lib\x64;;c:\tools\sdks\dx10\lib\x64
  LIBPATH=C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Windows\Microsoft.NET\Framework64\v2.0.50727;C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Windows\Microsoft.NET\Framework64\v2.0.50727;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\ATLMFC\LIB\amd64;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\LIB\amd64;
  LOCALAPPDATA=C:\Users\cltbld\AppData\Local
  LOG="c:\tmp\buildbot-startup.log"
  LOGNAME=cltbld
  LOGONSERVER=\\W64-IX-SLAVE79
  MACHTYPE=i686-pc-msys
  MAKE_MODE=unix
  MOZBUILDDIR=C:\mozilla-build\
  MOZILLABUILD=C:\mozilla-build\
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_MAXWINSDK=999999
  MOZ_MSVCVERSION=9
  MOZ_OBJDIR=obj-firefox
  MOZ_SIGN_CMD=python e:/builds/moz2_slave/elm-w32-ntly/tools/release/signing/signtool.py --cachedir e:/builds/moz2_slave/elm-w32-ntly/signing_cache -t e:/builds/moz2_slave/elm-w32-ntly/token -n e:/builds/moz2_slave/elm-w32-ntly/nonce -c e:/builds/moz2_slave/elm-w32-ntly/tools/release/signing/host.cert -H signing1.build.scl1.mozilla.com:9110 -H signing2.build.scl1.mozilla.com:9110 -H dev-master01.build.mozilla.org:8080
  MOZ_SYMBOLS_EXTRA_BUILDID=elm
  MOZ_TOOLS=C:\mozilla-build\\moztools-x64
  MOZ_UPDATE_CHANNEL=nightly-elm
  MSVC10EXPRESSKEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VCExpress\10.0\Setup\VC
  MSVC10KEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\10.0\Setup\VC
  MSVC6KEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\6.0\Setup\Microsoft Visual C++
  MSVC71KEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\7.1\Setup\VC
  MSVC8EXPRESSKEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VCExpress\8.0\Setup\VC
  MSVC8KEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\8.0\Setup\VC
  MSVC9EXPRESSKEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VCExpress\9.0\Setup\VC
  MSVC9KEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\9.0\Setup\VC
  MSVCEXPROOTKEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VCExpress
  MSVCROOTKEY=HKLM\SOFTWARE\Wow6432Node\Microsoft\VisualStudio
  MSYSTEM=MINGW32
  NUMBER_OF_PROCESSORS=4
  OLDPWD=c:/Users/cltbld
  OS=Windows_NT
  OSTYPE=msys
  PATH=C:\mozilla-build\nsis-2.46u;C:\mozilla-build\buildbotve\scripts;C:\mozilla-build\python27;C:\mozilla-build\msys\local\bin;c:\mozilla-build\wget;c:\mozilla-build\7zip;c:\mozilla-build\blat261\full;c:\mozilla-build\python;c:\mozilla-build\svn-win32-1.6.3\bin;c:\mozilla-build\upx203w;c:\mozilla-build\emacs-22.3\bin;c:\mozilla-build\info-zip;c:\mozilla-build\nsis-2.22;c:\mozilla-build\nsis-2.33u;c:\mozilla-build\nsis-2.46u;c:\mozilla-build\wix-351728;c:\mozilla-build\hg;c:\mozilla-build\python\Scripts;c:\mozilla-build\kdiff3;c:\mozilla-build\yasm;.;C:\mozilla-build\msys\local\bin;C:\mozilla-build\msys\mingw\bin;C:\mozilla-build\msys\bin;c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64;c:\Windows\Microsoft.NET\Framework64\v3.5;c:\Windows\Microsoft.NET\Framework64\v3.5\Microsoft .NET Framework 3.5 (Pre-Release Version);c:\Windows\Microsoft.NET\Framework64\v2.0.50727;c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\VCPackages;c:\Tools\msvs9\Common7\IDE;c:\Tools\msvs9\Common7\Tools;c:\Tools\msvs9\Common7\Tools\bin;c:\Tools\sdks\v7.0\bin\x64;c:\Tools\sdks\v7.0\bin\win64\x64;c:\Tools\sdks\v7.0\bin;c:\windows\System32;c:\windows;c:\windows\System32\Wbem;c:\mozilla-build\moztools-x64\bin;c:\mozilla-build\vim\vim72
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
  PDBSTR_PATH=/c/Program Files/Debugging Tools for Windows (x64)/srcsrv/pdbstr.exe
  POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_ARCHITEW6432=AMD64
  PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 30 Stepping 5, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=1e05
  PROGRAMDATA=C:\ProgramData
  PROGRAMFILES=C:\Program Files (x86)
  PROGRAMFILES(X86)=C:\Program Files (x86)
  PROGRAMW6432=C:\Program Files
  PROMPT=$P$G
  PS1=\[\033]0;$MSYSTEM:\w\007
\033[32m\]\u@\h \[\033[33m\w\033[0m\]
$ 
  PSMODULEPATH=C:\windows\system32\WindowsPowerShell\v1.0\Modules\
  PUBLIC=C:\Users\Public
  PWD=c:/Users/cltbld
  SDK2003SP1KEY=HKLM\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\8F9E5EF3-A9A5-491B-A889-C58EFFECE8B3
  SDK2003SP2KEY=HKLM\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\D2FF9F89-8AA2-4373-8A31-C838BF4DBBE1
  SDK61KEY=HKLM\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.1
  SDK6AKEY=HKLM\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.0A
  SDK6KEY=HKLM\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.0
  SDK7KEY=HKLM\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0
  SDKDIR=C:\Tools\sdks\v7.0\
  SDKMINORVER=0
  SDKROOTKEY=HKLM\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs
  SDKVER=7
  SESSIONNAME=Console
  SHELL=C:/mozilla-build/msys/bin/sh
  SHLVL=1
  SSH_AGENT_PID=2484
  SSH_AUTH_SOCK=C:/Users/cltbld/AppData/Local/Temp/ssh-lhYTaF2368/agent.2368
  SYMBOL_SERVER_HOST=dev-stage01.srv.releng.scl3.mozilla.com
  SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/
  SYMBOL_SERVER_SSH_KEY=/c/Users/cltbld/.ssh/ffxbld_dsa
  SYMBOL_SERVER_USER=ffxbld
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:\windows
  TEMP=C:/Users/cltbld/AppData/Local/Temp
  TEMPVC9DIR=C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
  TERM=cygwin
  TINDERBOX_OUTPUT=1
  TMP=C:/Users/cltbld/AppData/Local/Temp
  USERDOMAIN=W64-IX-SLAVE79
  USERNAME=cltbld
  USERPROFILE=C:\Users\cltbld
  VC10DIR=C:\Tools\msvs10\VC\
  VC9DIR=C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\
  VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC
  VS100COMNTOOLS=C:\Tools\msvs10\Common7\Tools\
  VS90COMNTOOLS=C:\Tools\msvs9\Common7\Tools\
  VSINSTALLDIR=C:\Tools\msvs9
  WIN64=1
  WINCURVERKEY=HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion
  WINDIR=C:\windows
  WINDOWSSDKDIR=C:\Tools\sdks\v7.0\
  WIX_351728_PATH=c:/mozilla-build/wix-351728

Here is the full log for the compile step:
http://people.mozilla.com/~armenzg/incoming/elm.staging.nightly.with.pgo.txt

Comment 37

[Jim Mathies [:jimm]]

I filed bug 809140 on the pgo issue.

Comment 38

[Armen [:armenzg]]

Attachment: run debug jobs on custom builders + enable pymake

I'm putting PGO out of the matrix until we can fix it.

This patch will fix the debug builders on Elm.

Comment 39

[Armen [:armenzg]]

Comment on attachment 678819 [details] [diff] [review]
run debug jobs on custom builders + enable pymake

4518140654c3 - this will become live once the buildbot masters are reconfigured.
I checked with jimm and this can wait until tomorrow morning (we prefer doing reconfigurations earlier in the day as they go smoother).

Comment 40

[Jim Mathies [:jimm]]

We can try pgo builds again, a fix for the last problem has landed on elm.

Comment 41

[Jim Mathies [:jimm]]

Maybe we should do this in three stages - 

1) get win debug using the custom builders
2) enable pymake and trigger builds on elm to test
3) enable pgo and trigger a build

Comment 42

[Armen [:armenzg]]

Hi jimm,
I will be enabling #1 & #2 at noon EST since we have got good results on staging.

I'm also testing #3 on staging and hopefully that will be enabled as well by then.

Comment 43

[Armen [:armenzg]]

Attachment: enable pgo

I already got r+ from coop on attachment 678721 [details] [diff] [review].
Once this proves to work on staging I will land it.

Comment 44

[Rail Aliiev [:rail]]

in production

Comment 45

[Armen [:armenzg]]

Comment on attachment 679141 [details] [diff] [review]
enable pgo

Landed as e1410b7ba829.
Tomorrow we will have a reconfiguration and I will trigger an Elm nightly to verify that it works.

Comment 46

[Jim Mathies [:jimm]]

It took forever but I finally have a successful local pgo build with that last checkin. So hopefully all will go well tomorrow.

Comment 47

[Armen [:armenzg]]

I will enable PGO this morning.

A side question, how does bug 794983 block us in here?

Comment 48

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #47)
> I will enable PGO this morning.
> 
> A side question, how does bug 794983 block us in here?

That work has landed on elm so consider it 'resolve fix', although we won't be marking the bug as resolved until elm's /widget/windows/winrt/* merges to mc. In the white board we have a tag for this - 'completed-elm'.

Comment 49

[Armen [:armenzg]]

PGO is now live. I have marked the builders to clobber and triggered a new nightly. It will take a while though.

Comment 50

[Armen [:armenzg]]

Anything left in here?

Are we just waiting to see if the Elm build has PGO working properly?

Comment 51

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #50)
> Anything left in here?
> 
> Are we just waiting to see if the Elm build has PGO working properly?

In the pgo nightly I see a strange error - 

PGOCVT : fatal error PG0001: An unexpected internal error was detected in source file 'f:\dd\vctools\compiler\utc\src\tools\pogo\cvtpgd\cvtpgd.cpp', line 800.
PGOCVT : fatal error PG0001: An unexpected internal error was detected in source file 'f:\dd\vctools\compiler\utc\src\tools\pogo\cvtpgd\cvtpgd.cpp', line 858. 

This might be a passive failure that's just getting picked up by tbpl. Need to test the builds to make sure they work.

Comment 52

[Jim Mathies [:jimm]]

PGO build look ok, so I think we can close this out.

Comment 53

[Armen [:armenzg]]

jimm, how soon will you know 100% that we're sticking with VS2010 and SDK 8?
Any reason not to start updating the rest of the win64 pool?

Comment 54

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #53)
> jimm, how soon will you know 100% that we're sticking with VS2010 and SDK 8?
> Any reason not to start updating the rest of the win64 pool?

We are going to stick with the 8.0 sdk and VS 2010 assuming all the testing we do of 8.0 sdk builds goes well. That's the next step in deciding whether or not we stick with this setup. My guess is we will. We certainly will not go back the vc 2012 unless something really ugly shows up, and based on the test results on elm it doesn't look like that will happen.

Which brings up an obvious question I can't answer - what are the steps we normally take in validating a new sdk for release builds? We should file appropriate bugs and kick this process off.

Comment 55

[Armen [:armenzg]]

Hi akeybl, lsblakk,
What is the process to request switching from current Windows 7.0 SDK to Windows 8.0 SDK?
Do you know?

Comment 56

[Alex Keybl [:akeybl]]

(In reply to Armen Zambrano G. [:armenzg] from comment #55)
> Hi akeybl, lsblakk,
> What is the process to request switching from current Windows 7.0 SDK to
> Windows 8.0 SDK?
> Do you know?

I think the right path forward would be an email to dev-platform that includes the proposal, reasoning for making the change, any known issues, and a request for any reasons to *not* move forward with the proposal.

Comment 57

[Armen [:armenzg]]

jimm, do you want to write that down and post it?

Thanks akeybl!

Comment 58

[Jim Mathies [:jimm]]

What's the complete set of step here though? There are some things that need to happen - announcing we plan to make this change is one, but we also need to validate the builds through qa testing as well. Which I would think we would want to do before we roll out the sdk to builders and possibly before we announce we're making the change to dev.planning.

cc'ing a few people who might know what steps we followed when we've done this in the past.

Comment 59

[(not currently active) Ted Mielczarek]

Are we planning on requiring SDK 8.0 as the base SDK for building Gecko, or will we support building with older SDKs?

In terms of what we should do here, as long as you announce it in dev.platform and verify that we don't break on WinXP we should be fine.

Comment 60

[Jim Mathies [:jimm]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #59)
> Are we planning on requiring SDK 8.0 as the base SDK for building Gecko, or
> will we support building with older SDKs?

Older sdks will be fine with --disable-metro.

> In terms of what we should do here, as long as you announce it in
> dev.platform and verify that we don't break on WinXP we should be fine.

Ok, sound good. I guess we can rely on our normal release train qa process then for 
testing?

Comment 61

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #57)
> jimm, do you want to write that down and post it?
> 
> Thanks akeybl!

Sure, let's move this to bug 774910 though.