772989 - After installing VS2012 on win64 machine I can't create normal win32 builds anymore

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Armen [:armenzg]]

Attachment: differences between a production slaves and a modified slaves

jimm, could you please help me with this?

I have attached the differences between a win64 slave w/o VS2012 (production) and a win64 slave w. VS2012 (staging).

I will try getting all the env variables exactly the same and see what happens.

The whole thing ends with this:
make[5]: Entering directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build'
make -C win32 libs
make[6]: Entering directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build/win32'
/e/builds/moz2_slave/elm-w32/build/obj-firefox/_virtualenv/Scripts/python.exe /e/builds/moz2_slave/elm-w32/build/config/pythonpath.py -I../../config /e/builds/moz2_slave/elm-w32/build/config/expandlibs_exec.py --depend .deps/crashinject.exe.pp --target crashinject.exe --uselist -- link -NOLOGO -OUT:crashinject.exe -PDB:crashinject.pdb  -LARGEADDRESSAWARE -NXCOMPAT -DYNAMICBASE -SAFESEH  -DEBUG -DEBUGTYPE:CV -DEBUG -OPT:REF    crashinject.obj ./module.res   kernel32.lib user32.lib gdi32.lib winmm.lib wsock32.lib advapi32.lib
LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt
make[6]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build/win32'
make[5]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox/build'
make[4]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox'
make[3]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox'
make[2]: Leaving directory `/e/builds/moz2_slave/elm-w32/build/obj-firefox'
make[1]: Leaving directory `/e/builds/moz2_slave/elm-w32/build'
make[6]: *** [crashinject.exe] Error 99
make[5]: *** [libs] Error 2
make[4]: *** [libs_tier_base] Error 2
make[3]: *** [tier_base] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1342032426.1342032448.17338.gz&fulltext=1

Comment 1

[Jim Mathies [:jimm]]

LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt

The most common causes of this are:

1) miss matched compiler / linker versions
2) building in an object dir with one version of the toolset and then rebuilding with another.
3) miss matched sdk / toolset

According to the build log the env is setup right for vs10:

Adding client.mk options from /e/builds/moz2_slave/elm-w32/build/.mozconfig:
    PROFILE_GEN_SCRIPT=$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/profileserver.py
    MOZ_MAKE_FLAGS=-j1
    export LIB=/c/tools/msvs10/vc/lib:/c/tools/msvs10/vc/atlmfc/lib:/c/tools/sdks/v7.0/lib:/c/tools/sdks/dx10/lib
    export LIBPATH=/c/tools/msvs10/vc/lib:/c/tools/msvs10/vc/atlmfc/lib
    export PATH=/c/tools/msvs10/Common7/IDE:/c/tools/msvs10/VC/BIN:/c/tools/msvs10/Common7/Tools...
    export INCLUDE=/c/tools/msvs10/vc/include:/c/tools/msvs10/vc/atlmfc/include:/c/tools/sdks/v7.0/include:/c/tools/sdks/v7.0/include/atl:/c/tools/sdks/dx10/include
    export WIN32_REDIST_DIR=/c/tools/msvs10/VC/redist/x86/Microsoft.VC100.CRT
    export MOZ_TOOLS=C:/mozilla-build/moztools

So I'm not sure what's going on here. If you can rdp into the slave, you might try running start-msvc10.bat and doing a build to be sure the vs2010 tools are working correctly. If that's the case then it must be something in the env of the automation shell.

Comment 2

[Armen [:armenzg]]

Attachment: error log with start-msvs10.bat

I get the same results when I run start-msvs10.bat.

jimm, would you be able to borrow a machine and try out?

I have tried running this:
export MOZ_OBJDIR=obj-firefox
make -f client.mk build &> ~/Desktop/1.6.errors.log.txt

Comment 3

[Armen [:armenzg]]

Attachment: differences between a production slaves and a modified slaves (3 more variables)

I am testing on staging a patch that adds the missing variables I saw on the logs:
                 'HG_SHARE_BASE_DIR': 'e:/builds/hg-shared',
                 'BINSCOPE': 'C:\Program Files (x86)\Microsoft\SDL BinScope\BinScope.exe',
+                'SDKVER': '7',
+                'WINDOWSSDKDIR': 'C:\\Tools\\sdks\\v7.0\\',
+                'SDKDIR': 'C:\\Tools\\sdks\\v7.0\\',
                 'PATH': "${MOZILLABUILD}buildbotve\\scripts;${PATH}",
even then we fail.

The only "extra" variable I see on the failing slaves is:
VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\

Comment 4

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #2)
> Created attachment 641594 [details]
> error log with start-msvs10.bat
> 
> I get the same results when I run start-msvs10.bat.
> 
> jimm, would you be able to borrow a machine and try out?


Sure, hook me up with access info, I'll take a look at this over the weekend.

Comment 5

[Jim Mathies [:jimm]]

So not sure how this slave got in the state it's in. The vs10 toolset is fine, except there's a resource building related tool called cvtres.exe that the linker uses to convert .res resource files over to COFF. On this slave the vs10 cvtres.exe is installed in:

C:\Tools\msvs10\VC\bin

This exe has an odd dependency on a .net library named MSVCR100_CLR0400.DLL. According to this social post this comes down with .NET 4.0.

http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/9e205d5c-2315-4a6a-95cc-92c5331aad09

Odd thing is I checked my local system and I have a different vs10 cvtres.exe that depends on the normal runtime dlls. 

So I'm curious, is this slave the same as our other builders, or did we set this machine up manually? I'm not sure why this vs10 install has this weird cvtres.exe in it but I seriously doubt it was modified via the 2012 install which installs the correct exe in it's tools folder.

In fact, I took the cvtres from vs 11, copied it over to vs10 tools folder, and the vs10 build succeeds.

Comment 6

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #5)
> 
> So I'm curious, is this slave the same as our other builders, or did we set
> this machine up manually? I'm not sure why this vs10 install has this weird
> cvtres.exe in it but I seriously doubt it was modified via the 2012 install
> which installs the correct exe in it's tools folder.
> 
I only installed vs2012 on the standard w64 slaves that we have.

> In fact, I took the cvtres from vs 11, copied it over to vs10 tools folder,
> and the vs10 build succeeds.
>

What was the exact command you typed to accomplish this?

Comment 7

[Jim Mathies [:jimm]]

(In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #6)
> (In reply to Jim Mathies [:jimm] from comment #5)
> > 
> > So I'm curious, is this slave the same as our other builders, or did we set
> > this machine up manually? I'm not sure why this vs10 install has this weird
> > cvtres.exe in it but I seriously doubt it was modified via the 2012 install
> > which installs the correct exe in it's tools folder.
> > 
> I only installed vs2012 on the standard w64 slaves that we have.
> 
> > In fact, I took the cvtres from vs 11, copied it over to vs10 tools folder,
> > and the vs10 build succeeds.
> >
> 
> What was the exact command you typed to accomplish this?

I did it from explorer but a cp command would accomplish the same. I'm not sure this is the solution though. Is there any way you could check the dependencies of cvtres.exe before and after a vc11 install? Maybe the vc11 install is playing into this somehow.

Comment 8

[Armen [:armenzg]]

Comparing a production slave pre-installation with one of the ones we have should do.

I will check when I come back if coop does not get to it.

Comment 9

[Jim Mathies [:jimm]]

(In reply to [:armenzg] - gone from Aug. 3rd to Aug. 27th from comment #8)
> Comparing a production slave pre-installation with one of the ones we have
> should do.
> 
> I will check when I come back if coop does not get to it.

This is interesting - see bug 774910. So I guess our slaves don't have the 7.1 sdk and hence don't have .net 4.0. That might explain this partially, although I still don't understand how the 7.0 sdk on this slave managed to get a resource compiler in it that was dependent on 4.0 .net.

My local system has all the sdks installed, so I wouldn't see this.

Comment 10

[Armen [:armenzg]]

It seems that cvtres.exe is the same on a production machine VS a production machine that got VS2012 installed. [1]

The vs11 file seems different than the one installed on vs10 [2].

To copy the file around we can do this:
runas /user:Administrator "cp 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\cvtres.exe' C:\Tools\msvs10\VC\bin"

So far, the build that I triggered is now compiling.

Should I try to install the SDK 7.1? rather than use the workaround?

#### 1)
C:\Tools\msvs10\VC\bin>dir cvtres.exe
 Volume in drive C is OSDisk
 Volume Serial Number is B050-D838

 Directory of C:\Tools\msvs10\VC\bin

03/18/2010  01:16 PM            31,048 cvtres.exe
               1 File(s)         31,048 bytes
               0 Dir(s)  35,849,699,328 bytes free

C:\Tools\msvs10\VC\bin>C:\mozilla-build\msys\bin\md5sum.exe C:\Tools\msvs10\VC\bin\cvtres.exe
\dfa8e7cdfc7a0e6673ec2459d494a67c *C:\\Tools\\msvs10\\VC\\bin\\cvtres.exe

#### 2)
C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin>dir cvtres.exe
 Volume in drive C is OSDisk
 Volume Serial Number is 94F5-6A6B

 Directory of C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin

07/26/2012  07:08 PM            42,432 cvtres.exe
               1 File(s)         42,432 bytes
               0 Dir(s)  30,024,310,784 bytes free

C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin>C:\mozilla-build\msys\bin\md5sum.exe cvtres.exe
b90e3c77ac49d767369dd3da98db26ce *cvtres.exe

Comment 11

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #10)
> Should I try to install the SDK 7.1? rather than use the workaround?
> 
Installing the 7.1 SDK did not work. The workaround is our only fix at the moment.

I wonder if an environment variable or registry indicates which cvtres to be used.

Comment 12

[Jim Mathies [:jimm]]

(In reply to Armen Zambrano G. [:armenzg] from comment #11)
> (In reply to Armen Zambrano G. [:armenzg] from comment #10)
> > Should I try to install the SDK 7.1? rather than use the workaround?
> > 
> Installing the 7.1 SDK did not work. The workaround is our only fix at the
> moment.
> 
> I wonder if an environment variable or registry indicates which cvtres to be
> used.

It's just one of the many files in the sdk the build process uses to link binaries.

So to recap on this:

we have win2008 builders that have:

1) vc2010/w the 7.0 sdk
2) we install a side-by-side installation of vc2012 w/8.0 sdk
3) cvtres.exe in the 7.0 sdk somehow becomes dependent on a non-existent system library.

This shouldn't too hard to track down by stepping through the vc2012 install process and tracking changes to cvtres before and after the install.

I can investigate this in detail. I'm a little wrapped up in mtro preview release work though, is this bug blocking anything major, or can it wait until after October?

Comment 13

[Jim Mathies [:jimm]]

err, "or can it wait until after *September*?"

Comment 14

[Armen [:armenzg]]

(In reply to Jim Mathies [:jimm] from comment #13)
> err, "or can it wait until after *September*?"

It can wait.

This would only block us when we want to deploy VS2012 to all win64 slaves.
For now I can do the workaround on the 5 slaves we have for elm.

Comment 15

[Armen [:armenzg]]

FTR I run this step:
cp 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\cvtres.exe' C:\Tools\msvs10\VC\bin
on all of our Elm dedicated slaves.

Comment 16

[Armen [:armenzg]]

We dropped the necessity to use VS2012.

Comment 17

[Frank Wein [:mcsmurf]]

Just fyi (maybe interesting for others that have the same problem): The workaround from Comment 15 worked for me on Windows 7 x64, too. I had Visual Studio 2010 installed and it was working just fine. Then I installed Visual Studio 2012 and my build broke trying to link vmwarerecordinghelper.dll. After copying cvtres.exe from Microsoft Visual Studio 11.0\VC\bin\amd64 to Microsoft Visual Studio 10.0\VC\bin (and overwriting the existing one) my build seems to work fine (maybe I should have copied the \bin version and not the \bin\amd64 version?). But it's still compiling atm, so I'll see later if the resulting build works fine.

Comment 18

[Curtis]

See http://support.microsoft.com/kb/2757355

cvtres.exe from .Net 4.0 RTM (VS 2010 RTM) had a dependency on msvcr100_clr0400.dll
.Net 4.0 SP1 apparently removes msvcr100_clr0400.dll

Updating VS2010 to SP1 updates cvtres.exe to a version that does not require msvcr100_clr0400.dll

Comment 19

[Jim Mathies [:jimm]]

(In reply to Curtis from comment #18)
> See http://support.microsoft.com/kb/2757355
> 
> cvtres.exe from .Net 4.0 RTM (VS 2010 RTM) had a dependency on
> msvcr100_clr0400.dll
> .Net 4.0 SP1 apparently removes msvcr100_clr0400.dll
> 
> Updating VS2010 to SP1 updates cvtres.exe to a version that does not require
> msvcr100_clr0400.dll

Thanks for tracking that down.