Closed Bug 562459 Opened 13 years ago Closed 12 years ago

Install Microsoft_VC80_DebugCRT_x86.msi on XP & Win7 test slaves (not Win7x64)

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: anodelman, Assigned: armenzg)

References

Details

(Whiteboard: [unittest])

Attachments

(3 files)

Add VC2005 Debug CRT for r3 XP slaves
2.53 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
[screenshot] "xpcshell.exe - The application failed to initialize properly (0xc0150004)."
119.67 KB, image/png
Details
output from depend walker
2.77 KB, text/plain
Details 
Currently get this error when running debug mochitest on win7:

ERROR: Cannot resolve reference Microsoft.VC80.DebugCRT,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8.
0.50727.762".

bc says:
i think if we did the manifest right it would package the dlls we need. but looking at the executables, the manifest was empty.

I would prefer not having to install visual studio on all talos slaves.
Blocks: 549458
Ted, is it intentional that we package msvc{p,m,r}80.dll on debug, while on opt we have mozcrt19.dll ? We have --enable-jemalloc in the debug mozconfig. There's no mozcrt19.dll (or mozcpp19.dll) in the debug package at all.
Figured the packaging out. At
 http://hg.mozilla.org/mozilla-central/file/c753325a40ff/browser/installer/package-manifest.in#l60
we include the Microsoft CRT files if MOZ_MEMORY is not defined, otherwise our own mozcrt19 and cpp dlls. 

The mozconfig for the debug build is
 http://hg.mozilla.org/build/buildbot-configs/file/45f88d2ae58e/mozilla2/win32/mozilla-central/debug/mozconfig
The --enable-jemalloc sets MOZ_MEMORY to 1, but --enable-tracemalloc disables it. AIUI we use tracemalloc to get bloat numbers, and also to detect leaks when running tests.
It's not going to help you either way, because the debug CRT is named differently. It's MSVCRTD80.dll. I'm not about to include that in the packaging manifest though, because Microsoft doesn't give you the rights to redistribute those files to other people, so I don't think we should even upload them to FTP.
We may be able to just copy the debug CRT DLLs to the SxS install folder on the Talos slaves. They get installed to something like c:\windows\winsxs\x86_microsoft.vc80.debugcrt...

You could try locating that directory on a build slave, copying it to the same place on a Talos slave and running a debug build. Hopefully there's not other magic at work (like registry keys).
I tried that and couldn't get it to work. Not to say someone else won't be able to though.
Ok, maybe try doing it in a cheesier fashion, just grab the contents of C:\Program Files\Microsoft Visual Studio 9.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC90.DebugCRT and stick it in c:\windows\system32 on the target machine?
comment #6, no love.

I really don't want to have to install the whole thing. :(
Yuck. So, one more thought, that would require a little bit of work, would be to copy all those files (the DLLs + the manifest) into the firefox directory after unzipping the build. We used to ship Firefox that way, so I know it will work.
Something that still needs to be tried.

bmoss proposed a fix for the missing visual studio environment involved a windows program that forces dlls to register themselves, could work by dropping the dlls we need into windows\system32 and then running that program (http://ss64.com/nt/regsvr32.html).
I think regsvr32 only works on COM components, not on regular DLLs, so I'm not sure it's what you need here.
Alice, are you actively working on this?
Assignee: nobody → anodelman
Priority: -- → P3
Whiteboard: [unittest]
Pushing to joduinn.
Assignee: anodelman → joduinn
Assignee: joduinn → armenzg
Putting *debug* unit tests for now on the side (P4) until I get some more air to breathe.
Priority: P3 → P4
I won't be able to touch this for a while. Check with me if you think I should pick this work ASAP over the other bugs I am already working on.
Assignee: armenzg → nobody
No surprise here, but this affects 64-bit Windows too. Missing MSVCR100D.dll  in this case.
No, that's only the release CRT.
When I try to run a debug build from Tinderbox on my own Windows box, I get exactly the same error as in comment 0:

Cannot resolve reference Microsoft.VC80.DebugCRT,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8.0.50727.762"

I'm using Windows 7 (64-bit) and I have MSVC 2008 Express Edition installed.  What else do I need to do in order to run debug builds from Tinderbox?
Version 8.00.50727.762 is MSVC 2005 SP1, I believe, so you'll need that version of the compiler toolchain installed for it to run.
No longer blocks: 549458
sayre: fyi, this is whats blocking us from starting unittest-on-debug-builds-on-minis.
If you want to hand me a Talos slave I can try fiddling with this and see if I can figure out a way forward.
(In reply to comment #21)
> If you want to hand me a Talos slave I can try fiddling with this and see if I
> can figure out a way forward.

I've sent Ted the login details for a Talos Win7 box.
Copying the contents of the "Microsoft.VC80.DebugCRT" dir into the app directory makes it possible to run the debug build. I'm going to see if I can find any alternative, but at least that's a fallback position.
Is the Debug.CRT stuff redistributable?
Okay, I've got a solution. Turns out Microsoft has a document describing this exact scenario:
http://msdn.microsoft.com/en-us/library/aa985618%28v=VS.80%29.aspx

except the simple solution there (msiexec /i whatever.msm) doesn't actually work. But, there's a free msm2msi converter available that produces a msi package that can be installed and does exactly what we need. Here's what I did:

1) Download msm2msi from http://www.ethalone.com/articles/61-using-windows-installer-merge-modules-in-ghost-installer-studio.html
2) Run msm2msi.exe "C:\Program Files\Common Files\Merge Modules\Microsoft_VC80_DebugCRT_x86.msm", which produces Microsoft_VC80_DebugCRT_x86.msi in the same "Merge Modules" directory.
3) Install the resulting msi on the Talos slave (I just used msiexec /i Microsoft_VC80_DebugCRT_x86.msi)

I was then able to successfully run a debug Firefox build from the tinderbox-builds/mozilla-central-win32-debug directory on the Talos slave.
(In reply to comment #24)
> Is the Debug.CRT stuff redistributable?

No, it's explicitly not redistributable, unfortunately.
(In reply to comment #25)
> Here's what I did:

I forgot to mention, steps 1 and 2 were done on a build slave machine (but could be done on any machine with Visual C++ 2005 installed).
These are real great news.
ted thanks a lot for your help over here :)
I would have wasted a lot of time until I figured this out.
Assignee: nobody → armenzg
Priority: P4 → P3
Summary: be able to run debug packaged unit tests without installing visual studio (windows only) → Install Microsoft_VC80_DebugCRT_x86.msi on Rev3 Windows machines
Priority: P3 → P2
Depends on: 592353
This worked on staging-opsi.
Attachment #470858 - Flags: review?(ccooper)
Attachment #470858 - Flags: review?(ccooper) → review+
Filed bug 592806 to track that we can't run unit tests on XP slaves.

Putting on the side until it gets resolved.
Depends on: 592806
Priority: P2 → P4
For reference:
http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/Microsoft_VC80_DebugCRT_x86.msi

ab9907210c0f95733cb8465b4dfa28d8  Microsoft_VC80_DebugCRT_x86.msi
Priority: P4 → P5
Priority: P5 → P3
Status: NEW → ASSIGNED
Priority: P3 → P2
Priority: P2 → P3
Can you also post instructions for getting the correct version of the debug crt on developer machines?  I need to use Tinderbox builds locally in order to develop code for fuzzing jobs that will eventually run on the test pool.
You should be able to follow the steps in comment 25, assuming you have access to a machine with Visual C++ 2005 installed.
(Can I just install Visual C++ 2005 in addition to the newer version I already have?)
That would also work, yeah. The msi trick described there is much simpler, though, assuming you have access to a machine with VC2005. (Plus once you make the MSI once, you can easily install it on as many machines as you need .)
Armen, can you post your Microsoft_VC80_DebugCRT_x86.msi somewhere where I can get it, perhaps in http://fs/IT/Microsoft/ ?
I have emailed Jesse the redistributable for testing purposes.

To note that this has not been deployed to our testing machines and nothing has happened since the beginning of September. Hopefully I will get back to this before the end of November.
Priority: P3 → P4
I would like to get back to this and get it out of the way this following 2-3 weeks.
Priority: P4 → P2
I am trying to get this done for the Win7 machines (instead of the XP machines) as this will get us unblocked for running debug unit tests on the minis.

XP will come later.

To summarize what is going on:
* To run debug unit tests on the minis we need to run the debug build
* To run the debug build we need to install Microsoft_VC80_DebugCRT_x86.msm as mentioned in comment 25 otherwise we get a "sige-by-side" error [1]
* To install the Debug CRT through the command line and/or OPSI we need an msi
* To install it through the command line we have to run the following (cltbld has admin privileges):
> runas /user:cltbld "msiexec /quiet /i Microsoft_VC80_DebugCRT_x86.msi"
* To allow that command to succeed *I* believe we have to disable UAC [2] otherwise we can't succeed:
> runas /user:cltbld "C:\Windows\System32\cmd.exe /k %windir%\System32\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f"
* For the previous to work we need to give full control to the cltbld user [3], otherwise, I get denied to change the registry.

I am blocked right now.
* I don't know how to install the msi without disabling UAC
* I can't disable UAC without giving full control to cltbld and that requires using VNC/RemoteDesktop (as I wanted to avoid)

For now, will research if we can grant full control through the command line [3]
Worst case scenario I will have to deploy this manually on each windows machine.

[1] "The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail" http://www.vistax64.com/vista-performance-maintenance/222177-side-side-configuration-incorrect-every-program.html
[2] http://www.recipester.org/Recipe:Enable_or_Disable_UAC_From_the_Command_Line_in_Windows_7_37930745
[3] http://www.raymond.cc/blog/archives/2009/03/05/full-control-permission-to-delete-or-edit-restricted-windows-registry/
I am considering deploying the msi manually through VNC in all win7 slaves. A batch at a time.
This has been deployed to the win7 testing slaves:
* cd Desktop; wget http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/Microsoft_VC80_DebugCRT_x86.msi
# sha1sum ab9907210c0f95733cb8465b4dfa28d8  Microsoft_VC80_DebugCRT_x86.msi
* through VNC:
  * double click the msi
  * Accept UAC prompt
  * rm msi

These are the slaves that don't have the CRT deployed:
* w7-001 loaned to sdwilsh - bug 609050
* w7-012 needs reboot - bug 614821
* w7-024 out of the pool - bug 602312
* w7-040 host non-existant?? - bug 614821
* w7-ref I can't log in into it

Documented:
https://wiki.mozilla.org/ReferencePlatforms/Test/Win7#Microsoft_VC80_DebugCRT_x86.msi

TODO
* check-in file into private repos
Depends on: 609050, 602312, 614821
Depends on: 614862
Blocks: 614955
Blocks: 614956
(In reply to comment #26)
> (In reply to comment #24)
> > Is the Debug.CRT stuff redistributable?
> 
> No, it's explicitly not redistributable, unfortunately.

That really sucks, Tinderbox Debug builds are practically useless. :(
Sorry, you'll have to complain to Microsoft, there's nothing we can do about it. You can install their free compiler and build your own.
(In reply to comment #41)
> TODO
> * check-in file into private repos

Done.
(In reply to comment #41)
> # sha1sum ab9907210c0f95733cb8465b4dfa28d8  Microsoft_VC80_DebugCRT_x86.msi

This is an MD5 sum. talos-r3-w7-ref is done per steps in comment #41.
Thanks Nick!
Depends on: 618518
Priority: P2 → P3
Whiteboard: [unittest] → [unittest] To be completed with bug 614955
Picking this up again.
Priority: P3 → P2
I have installed the DebugCRT and tried to run various types of unit tests.

All of them fail to start with the following message:
"xpcshell.exe - The application failed to initialize properly (0xc0150004)."

Any ideas on what is going on?

I will look deeper into it early next week.

I am also looking at this article but I don't know how does it apply if I installed the DebugCRT:
http://blogs.msdn.com/b/heaths/archive/2007/02/05/visual-studio-2005-service-pack-1-rollback-breaks-some-applications.aspx
I need real help in here. I don't know what the attached depend walker output should be telling me to look for.

These are the steps that I follow:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1297891792/firefox-4.0b12pre.en-US.win32.zip
unzip -o firefox-4.0b12pre.en-US.win32.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1297891792/firefox-4.0b12pre.en-US.win32.tests.zip
unzip -o firefox-4.0b12pre.en-US.win32.tests.zip

At this point my problems start. I can substitute running the unit tests to just trying bin/xpcshell.exe (Note that I have to add the firefox/ directory to our PATH otherwise you will get can't find xpcom.dll message).

There are 3 problems that I have but I have only been able to figure out two of them:

Solution to 1st problem
########################
I placed the Microsoft.VC80.DebugCRT folder (from a build machine) into the bin/ folder that comes from unzipping the tests package to avoid the following message:
"This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem"

Solution to 2nd problem
#######################
Once the DebugCRT directory is placed under the bin/ directory I have to put the manifest file under firefox/  (Microsoft.VC80.DebugCRT.manifest) to avoid the following message:
"This application failed to initialize properly (0xc0150002)"

3rd problem
###########
The previous does not seem sufficient as I get the following:
"xpcshell.exe - Application Error" - The application failed to initialize properly (0xc0000034). Click on OK to terminate the application"

FTR: python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --symbols-path=symbols --certificate-path=certs --autorun --close-when-done --console-level=INFO --total-chunks=5 --this-chunk=1 --chunk-by-dir=4

Created dev.builds thread http://groups.google.com/group/mozilla.dev.builds/browse_thread/thread/37032380988633d1#
Armen, if you can get me a Windows XP slave to poke at, I can take a look at this on Monday.
Depends on: 634992
Thanks ted, I have filed bug 634992 to grant you access to our VPN so you can have access to a machine.
Priority: P2 → P3
Whiteboard: [unittest] To be completed with bug 614955 → [unittest] need help from ted
Depends on: 635267
I talked to Armen on IRC, but I'll record it here for posterity. I logged into this slave, and installed the Debug CRT MSI that he had left on the Desktop (by double-clicking). I downloaded a build + test package from http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1298468457/ , unzipped them both to c:/temp, and was able to successfully run Mochitest.

Armen's going to check some things, I'm not sure if I missed something or if he was testing a different scenario.
I have verified what ted says.

I don't why the DebugCRT does not install anything (C:\Windows\WinSxS) on the two staging slaves I worked on.

I will look next week into how to deploy this through OPSI.

thanks ted for your help!
Depends on: 636471
Priority: P3 → P2
Whiteboard: [unittest] need help from ted → [unittest]
Comment on attachment 470858 [details] [diff] [review]
Add VC2005 Debug CRT for r3 XP slaves

The Debug CRT got deployed to:
http://hg.mozilla.org/build/opsi-package-sources/rev/9126788ed78a

This also got documented in here:
https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Microsoft_Visual_C.2B.2B_2005_Non-Redistributable_Debug_CRT_.28x86.29

NOTE: I have deployed to the first 20 XP slaves. If find no major problem I will mark the remaining slaves to pick up the change. FTR I am not doing manual reboots of slaves until I figure out what is going on with the new 3 masters coming up.
Attachment #470858 - Flags: checked-in+
(In reply to comment #54)
> Comment on attachment 470858 [details] [diff] [review]
> Add VC2005 Debug CRT for r3 XP slaves
> 
> The Debug CRT got deployed to:
> http://hg.mozilla.org/build/opsi-package-sources/rev/9126788ed78a
> 
> This also got documented in here:
> https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Microsoft_Visual_C.2B.2B_2005_Non-Redistributable_Debug_CRT_.28x86.29
> 
> NOTE: I have deployed to the first 20 XP slaves. If find no major problem I
> will mark the remaining slaves to pick up the change. FTR I am not doing manual
> reboots of slaves until I figure out what is going on with the new 3 masters
> coming up.

buildbot-master4 is out for hardware repair, buildbot-master5 and 6 are coming online now.
I have marked the remaining XP slaves to receive this package.
I can see this being deployed properly under C:\Windows\WinSxS

03/07/2011 01:11 PM <DIR> x86_Microsoft.VC80.DebugCRT_1fc8b3b9a1e18
e3b_8.0.50727.762_x-ww_5490cd9f
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
cltbld@production-opsi:~/opsi-binaries$ cvs add -kb vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi 
cvs add: scheduling file `vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi' for addition
cvs add: use 'cvs commit' to add this file permanently
cltbld@production-opsi:~/opsi-binaries$ cvs commit -m "Bug 562459 - Install Microsoft_VC80_DebugCRT_x86.msi on Rev3 Windows machines. r=coop"
RCS file: /mofo/opsi-binaries/vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi,v
done
Checking in vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi;
/mofo/opsi-binaries/vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi,v  <--  Microsoft_VC80_DebugCRT_x86.msi
initial revision: 1.1
done

talos-r3-xp-ref is also updated.

Nothing left to be done.
I will assume that this package have not been installed on the tool staging boxes?
Alice the Debug CRT is on fs under IT/Microsoft.
Armen - does this bug being fixed mean we can file a bug to turn on debug tests for w764?  http://hg.mozilla.org/build/buildbot-configs/file/3c3ff08406a5/mozilla-tests/config.py#l274 seems to suggest this would be the case.  You know the most about this, can you confirm?
This was not deployed to the Win7x64 machines so we should not enable it yet.
Summary: Install Microsoft_VC80_DebugCRT_x86.msi on Rev3 Windows machines → Install Microsoft_VC80_DebugCRT_x86.msi on XP & Win7 test slaves (not Win7x64)
(In reply to comment #62)
> This was not deployed to the Win7x64 machines so we should not enable it yet.

good to know - is there a bug tracking deploying this to win764 machines that we can update that comment with?
No, there isn't.
Feel free to file one but I would not bother until we have the win64 build slaves up which is going to take quite a while.

You could also change it to 'debug_unittest_suites':[] and get rid of lines 274 & 275.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.