Closed Bug 529938 Opened 15 years ago Closed 14 years ago

Install DirectX SDK on buildbots

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Platform:
x86
Windows 7
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bas.schouten, Assigned: rail)

References

Details

(Whiteboard: [buildslaves])

Attachments

(2 files, 2 obsolete files)

DirectX 10 SDK OPSI package [tested]
3.81 KB, patch
bhearsum
: review+
rail
: checked-in+
Details | Diff | Splinter Review
buildbot-startup OPSI package
1.91 KB, patch
bhearsum
: review+
rail
: checked-in+
Details | Diff | Splinter Review 
Since it appears inevitable that we will start using DirectX in the relatively near future. The DirectX SDK should be installed on the windows buildbots.

Latest DXSDK is currently at: http://www.microsoft.com/downloads/details.aspx?FamilyID=b66e14b8-8505-4b17-bf80-edb2df5abad4&displaylang=en#dx
Thanks for filing this in advance. Is there any way you can be more precise than "relatively near future"? Are we talking days, weeks, months?
(In reply to comment #1)
> Thanks for filing this in advance. Is there any way you can be more precise
> than "relatively near future"? Are we talking days, weeks, months?

For the try build bots we could already use it at the moment, since I have a patch using it already. I expect it to take a while to hit the actual repositor though, so definitely atleast weeks there. Just want to give notice well in advance.
(In reply to comment #2)
> (In reply to comment #1)
> > Thanks for filing this in advance. Is there any way you can be more precise
> > than "relatively near future"? Are we talking days, weeks, months?
> 
> For the try build bots we could already use it at the moment, since I have a
> patch using it already. I expect it to take a while to hit the actual repositor
> though, so definitely atleast weeks there. Just want to give notice well in
> advance.

OK. Thanks again for the advance notice, we'll try to get to this soon.
Armen said he is interested in taking this on, but not for a couple of weeks at least.
Assignee: nobody → armenzg
Putting back into the pool. Too much going on right now.
Assignee: armenzg → nobody
I might do this while I do a few other OPSI things.
Assignee: nobody → bhearsum
Blocks: 527707
The patch that depends on this is now < 2 weeks from being ready to land, as an update here.  Do we have a sense of when we should expect to see the DX SDK deployed on the build slaves?  I'm getting a lot of pokes about the D2D backend landing. :)
(In reply to comment #7)
> The patch that depends on this is now < 2 weeks from being ready to land, as an
> update here.  Do we have a sense of when we should expect to see the DX SDK
> deployed on the build slaves?  I'm getting a lot of pokes about the D2D backend
> landing. :)

Right now, we're wrapping up Q4 stuff. This is next on the list. I know its a lot closer to your 2 week planned landing then is ideal, but this seems right priority. Let me know if I'm missing something?

(Also, thanks for the advance headsup - this is really helpful for our planning!)
Assignee: bhearsum → nobody
Component: Release Engineering → Release Engineering: Future
So first 2 weeks of January?

(RE:Future's description makes the move of component worrying to those of us waiting for this to push a big gfx improvement out: "For longer term projects that have been agreed should be done, but have no immediate plans to so. These are not be part of the regular recurring triage. Advanced planning and placeholder goals for next quarter also go here.".  Can you confirm that that's not the spirit in which you moved it?)
I tried building on tryserver with the following configure.in file:
http://hg.mozilla.org/try/file/9b5aae125312/configure.in

It seemed to build fine (though I'm not certain d2d actually got build in)

The build is here: 
https://build.mozilla.org/tryserver-builds/jmuizelaar@mozilla.com-try-9b5aae125312/ if anyone wants to check if it actually has d2d.
Oh its most definitely in there.

Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.3a1pre) Gecko/20100108 Minefield/3.7a1pre ID:20100108161953
Confirmed. It is, so does this mean the try servers have the DX SDK now?
No, I believe it means that you don't need the DX SDK to build. i.e. a recent platform SDK is sufficient.
(In reply to comment #13)
> No, I believe it means that you don't need the DX SDK to build. i.e. a recent
> platform SDK is sufficient.
Looking at the platform SDK. There seem to be the files we need in there in the Windows 7 SDK, which is great ofcourse! My build stuff should check for dxsdkver.h, so I'm surprised the configure doesn't bailout though.
I took out the check for dxsdkver.h before testing on try. I guess this means this bug is resolved then?
Since the Windows 7 SDK is required I think I currently see no reason why we'd need the DirectX SDK.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → INVALID
Just to be crystal clear: We don't need to install the DirectX SDK because what we need is included in the Windows 7 one?
(In reply to comment #17)
> Just to be crystal clear: We don't need to install the DirectX SDK because what
> we need is included in the Windows 7 one?
As far as we can tell at this point. Yes. D3DX libraries aren't, but we don't want to use those anyway since they require shipping to the end user.
Moving closed Future bugs into Release Engineering in preparation for removing the Future component.
Component: Release Engineering: Future → Release Engineering
I am (sadly) going to have to reopen this bug... we'd like to build ANGLE as part of our build, and it relies on d3dx which is only available in the DirectX sdk. The latest DX SDK is now June '10, at http://www.microsoft.com/downloads/en/details.aspx?FamilyID=3021D52B-514E-41D3-AD02-438A3BA730BA .

Note that we'll need to both install it as well as add the lib/include dirs to the LIB/INCLUDE env vars.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
(In reply to comment #20)
> The latest DX SDK is now June '10, at
> http://www.microsoft.com/downloads/en/details.aspx?FamilyID=3021D52B-514E-41D3-AD02-438A3BA730BA
> However, Visual Studio 2005 will no longer be supported.
We are still using VS2005 for our official build.
Do we have to use an older DXSDK or migrate to VS2008? Note that we can't migrate to VS2010 unless we drop support for Win2k.
We have other blockers for migrating to VC2010 anyway, which aren't going to get fixed in time for FF4.
An older version of the SDK is probably fine, but we should figure out what "Visual Studio 2005 will no longer be supported" means -- if it means that the DX SDK tools won't work in VS2005, that's perfectly fine; the only thing we need from this are .h files and .lib files.  If for some reason the .lib files aren't compatible, that's a bigger problem.

I have vs2005 installed on a machine here, I'll check and see what happens.
Making .libs that don't work with VS2005 and work with VS2010 is very difficult, by my understanding.  I like your testing-it plan nonetheless!
Yeah, but making header files that don't work with a 5 year old compiler is pretty easy!
:(

dxguid.lib(d3dxguid.obj) : fatal error LNK1103: debugging information corrupt; recompile module
So, the February 2010 dx sdk is the last one that supports VS 2005 -- http://www.microsoft.com/downloads/en/details.aspx?FamilyID=2c7da5fb-ffbb-4af6-8c66-651cbd28ca15&displaylang=en

The only thing that needs to be installed is 'DirectX Headers and Libs', everything else should be checked off.  It might be worth installing 'DirectX Redistributable Files' as well, in case we need to go that route for packaging.  Docs, runtime, symbol files, etc. can be turned off.

However, the downside of the Feb 2010 SDK is that it does -not- support VS 2010.  There isn't a SDK that supports both 2005 and 2010; so something to keep in mind for when we upgrade.
(In reply to comment #27)
> So, the February 2010 dx sdk is the last one that supports VS 2005 --
> http://www.microsoft.com/downloads/en/details.aspx?FamilyID=2c7da5fb-ffbb-4af6-8c66-651cbd28ca15&displaylang=en
> 
> The only thing that needs to be installed is 'DirectX Headers and Libs',
> everything else should be checked off.  It might be worth installing 'DirectX
> Redistributable Files' as well, in case we need to go that route for packaging.
>  Docs, runtime, symbol files, etc. can be turned off.

1) Is there a command line installer, or a specific list of what exact files should go where?  I ask because we'd like to have configuration management drive installing this on all slaves identically - its faster (and safer) then installing manually on each individual build slave.

2) is this needed for FF4.0?

3) aiui, this is to be installed on build (2003server, 2008server) machines only, and is not needed on the test (xp, win7, win64) machines.


> However, the downside of the Feb 2010 SDK is that it does -not- support VS
> 2010.  There isn't a SDK that supports both 2005 and 2010; so something to keep
> in mind for when we upgrade.
good to know.

Are there any binary compatibility concerns for this when we do upgrade?
(In reply to comment #28)
> (In reply to comment #27)
> > So, the February 2010 dx sdk is the last one that supports VS 2005 --
> > http://www.microsoft.com/downloads/en/details.aspx?FamilyID=2c7da5fb-ffbb-4af6-8c66-651cbd28ca15&displaylang=en
> > 
> > The only thing that needs to be installed is 'DirectX Headers and Libs',
> > everything else should be checked off.  It might be worth installing 'DirectX
> > Redistributable Files' as well, in case we need to go that route for packaging.
> >  Docs, runtime, symbol files, etc. can be turned off.
> 
> 1) Is there a command line installer, or a specific list of what exact files
> should go where?  I ask because we'd like to have configuration management
> drive installing this on all slaves identically - its faster (and safer) then
> installing manually on each individual build slave.

No command line installer, but if you install the SDK in, say, c:\dxsdk, we need c:\dxsdk\include, c:\dxsdk\lib, and c:\dxsdk\redist.  I can zip those up and put them somewhere if that's easier.

> 2) is this needed for FF4.0?

Unfortunately, yes.

> 3) aiui, this is to be installed on build (2003server, 2008server) machines
> only, and is not needed on the test (xp, win7, win64) machines.

Correct, just on build machines.

> > However, the downside of the Feb 2010 SDK is that it does -not- support VS
> > 2010.  There isn't a SDK that supports both 2005 and 2010; so something to keep
> > in mind for when we upgrade.
> good to know.
> 
> Are there any binary compatibility concerns for this when we do upgrade?

Shouldn't be; we'll just need to install a more recent SDK along with vs2010.  They can coexist even, so we can install this in a dxsdk-vs2005 directory and the newer thing later in a dxsdk-vs2010 dir.  Just have to make sure the INCLUDE/LIB paths point to the right thing.
(In reply to comment #26)
> :(
> dxguid.lib(d3dxguid.obj) : fatal error LNK1103: debugging information corrupt;
> recompile module
I was able to build the tree using VS2005 + DX SDK June 2010 with "BUILD_ANGLE = 1" enabled. Just to be sure, did you install a hotfix from KB949009?
http://support.microsoft.com/kb/949009
I'll take care of this.
Assignee: nobody → rail
Status: REOPENED → ASSIGNED
Priority: -- → P2
Attached patch DirectX 10 SDK OPSI package (obsolete) — — Splinter Review 
> No command line installer, but if you install the SDK in, say, c:\dxsdk, we
> need c:\dxsdk\include, c:\dxsdk\lib, and c:\dxsdk\redist.  I can zip those up
> and put them somewhere if that's easier.

vlad, could you confirm that the following command creates the needed layout:

7z.exe x DXSDK_Feb10.exe -aoa -o"c:\tmp" DXSDK\Lib DXSDK\Include DXSDK\Redist

It should extract the files to c:\tmp\DXSDK\{Lib,Include,Redist} directories.

I installed SDK using both methods (7z and setup.exe). The only difference (diff -Nr) is c:\tmp\DXSDK\system directory, which contains runtime stuff and according to your message (quoted above) we shouldn't care about it.

OPSI package is attached.
Attachment #482516 - Flags: review?(bhearsum)
How about registry changes ? I'm not sure we need that if we're setting the environment explicitly, but we might if we let MozillaBuild do the detection.
I believe that we need HKLM/Software/Microsoft/DirectX SDK/ and children, in which case we want to install to a real path and not \tmp, so that it all lines up right.
(That said, I don't see any support in MozillaBuild for autodetecting the DXSDK path, which we probably need before we make it mandatory for developers.)
(In reply to comment #32)
> Created attachment 482516 [details] [diff] [review]
> DirectX 10 SDK OPSI package
> 
> > No command line installer, but if you install the SDK in, say, c:\dxsdk, we
> > need c:\dxsdk\include, c:\dxsdk\lib, and c:\dxsdk\redist.  I can zip those up
> > and put them somewhere if that's easier.
> 
> vlad, could you confirm that the following command creates the needed layout:
> 
> 7z.exe x DXSDK_Feb10.exe -aoa -o"c:\tmp" DXSDK\Lib DXSDK\Include DXSDK\Redist

Yep, that's the correct set of stuff.

For registry changes, easiest thing would probably be to teach mozilla build to get the dx path out of the registry, and then check c:\dxsdk or some known location (or check an env var, which is easy to set).  That way the install process can still be simple.

(In reply to comment #35)
> (That said, I don't see any support in MozillaBuild for autodetecting the DXSDK
> path, which we probably need before we make it mandatory for developers.)

Doesn't have to be mandatory, since the only thing that depends on it is ANGLE -- if you don't have it, we can just skip building the ANGLE libs like we do today.
(In reply to comment #35)
> (That said, I don't see any support in MozillaBuild for autodetecting the DXSDK
> path, which we probably need before we make it mandatory for developers.)

Indeed, MozillaBuild only knows about compilers and Windows SDKs.
* Added registry entries
* Install path = d:\sdks\dx10
Attachment #482516 - Attachment is obsolete: true
Attachment #482794 - Flags: review?(bhearsum)
Attachment #482516 - Flags: review?(bhearsum)
Comment on attachment 482794 [details] [diff] [review]
DirectX 10 SDK OPSI package [tested]

This looks correct, I'd like to see a build run against this install before we roll it out to everything though.
Attachment #482794 - Flags: review?(bhearsum) → review+
Vlad (or somebody else), could you provide instructions and mozconfig for a test build?
Yep, you can just do a normal build but set BUILD_ANGLE=1 in the environment before running make.  (This only affects the gfx/angle subdir.)  You should end up with libEGL.dll and libGLESv2.dll in the dist/bin directory.
Oh, that assumes that c:\sdks\dx10\include is in the INCLUDE env var, and c:\sdks\dx10\libs\x86 is in the LIB env var.
Build went fine, libEGL.dll and libGLESv2.dll seen in dist/bin and gfx/angle/angle-build directories.

Could you also sanity check the build: http://people.mozilla.org/~raliiev/firefox-4.0b8pre.en-US.win32.installer.exe ?
I can, but note that libEGL.dll/GLESv2.dll haven't yet been added to the install manifests, so they're unlikely to actually be present inside that installer.  Also, there are no code changes other than building the two DLLs, so there should be no differences in the build result with/without the SDK..
(In reply to comment #44)
> I can, but note that libEGL.dll/GLESv2.dll haven't yet been added to the
> install manifests, so they're unlikely to actually be present inside that
> installer.

Yeah, just un7zipped the exe, there is no libEGL.dll/GLESv2.dll.
Comment on attachment 482794 [details] [diff] [review]
DirectX 10 SDK OPSI package [tested]

http://hg.mozilla.org/build/opsi-package-sources/rev/28b9f1059916
Attachment #482794 - Flags: checked-in+
Try build slaves (try-w32-slave{01..36}, except try-w32-slave05, which is not listed in OPSI) should get this package in a couple of days after next reboot. I will check the status today and tomorrow and update the installation status.

After successfully testing in try we can deploy this package in production as well.
Awesome!  Let me know when it's deployed and I'll kick off a build with the various build bits set.
Please keep an eye on the nagios alerts, the try-slave-NN vm's should be much the same as the ones you've been having disk space issues with.
Vlad,

The slaves were very silent, so I decided to speed up DX10 SDK install. 

I gracefully shut down and rebooted the slaves, then checked if the package has been deployed without unexpected effects. The overall process went very smooth and took 1-2 minutes to install (per slave), even though the exe file is huge (555MB) and served over SMB.

At the moment all _try_ slaves have DX10 SDK installed to d:\sdks\dx10sdk. Feel free to play around it, but don't forget to explicitly set INCLUDE and LIB, because http://hg.mozilla.org/build/buildbotcustom/file/tip/env.py "knows" nothing about the new sdk.

Good luck!
Great!  I'll give it a shot.

Also, 555MB :o  That seems way too large, did something wrong sneak in?  lib+include+redist are only about 125MB on-disk uncompressed; the entire full install of the sdk is about 1.1GB, with 780MB taken up by the 'samples' dir.
555MB is not an installed size, but installer's size.
Oh, misunderstood -- didn't realize we were starting with Microsoft's actual sdk, thought we were creating our own 7z installer with just the bits we need.
(In reply to comment #52)
> 555MB is not an installed size, but installer's size.

Yeah, installer size: 555M, installed size: 127M

(In reply to comment #53)
> thought we were creating our own 7z installer with just the bits we need.

I used pristine copy to eliminate intermediate steps, so we can easily reuse new upstream versions (hopefully :) ).
Vlad, hey!

Had you chance to test the SDK? Should we install it on production machines as well?
Yep, working on testing it for the past little while.  I'm running into a compile issue, that I finally got around to figuring out how to get the logs out (they're hidden in a MSVC log file).  Don't know what the issue is yet, will comment as soon as I know -- hopefully later today.
So, not having INCLUDE/LIB set correctly in the buildmaster is making it hard to test this -- I'm trying http://hg.mozilla.org/try/rev/47d70938f0ed which does have $LIB set to:

D:\sdks\v7.0\\lib;D:\msvs8\VC\ATLMFC\LIB;D:\msvs8\VC\LIB;D:\msvs8\VC\PlatformSDK\lib;D:\msvs8\SDK\v2.0\lib;;D:\mozilla-build\\atlthunk_compat;d:\sdks\dx10sdk\lib\x86

at least as far as that 'echo' is concerned.  But the build is failing to find d3d9x.h and dxguid.lib.  Can you confirm that d:\sdks\dx10sdk\lib\x86 and d:\sdks\dx10sdk\include has the right files?  If it does, I have no idea how to set env vars in a way that are passed down correctly from the makefile, and it would be easier to just add then to env.py directly.
dxguid.lib is in lib\x86, but there is no d3d9x.h in \include. Maybe you misspelled it? I can see d3dx9.h and d3d9.h, but no d3d9x.h...

In any case I'll try to run a build using your revision in staging environment with a slightly tuned INCLUDE and LIB set by buildbot (see: http://hg.mozilla.org/build/buildbotcustom/file/552b02de6346/env.py#l58). Be back with the results later (I hope today).
I tried to build your revision and it failed with the following output:

=== Building ANGLE via devenv.exe === d:\sdks\v7.0\\lib;d:\msvs8\VC\ATLMFC\LIB;d:\msvs8\VC\LIB;d:\msvs8\VC\PlatformSDK\lib;d:\msvs8\SDK\v2.0\lib;;D:\mozilla-build\\atlthunk_compat;d:\sdks\dx10sdk\lib\x86
TranslatorGLSL.cpp
=== Building ANGLE via devenv.exe === d:\sdks\v7.0\\lib;d:\msvs8\VC\ATLMFC\LIB;d:\msvs8\VC\LIB;d:\msvs8\VC\PlatformSDK\lib;d:\msvs8\SDK\v2.0\lib;;D:\mozilla-build\\atlthunk_compat;d:\sdks\dx10sdk\lib\x86
cp: cannot create regular file `angle-build/src/ANGLE.sln': No such file or directory
cp: cannot create regular file `angle-build/src/build_angle.gyp': No such file or directory
cp: cannot create directory `angle-build/src/common': No such file or directory
cp: cannot create directory `angle-build/src/compiler': No such file or directory
cp: cannot create directory `angle-build/src/libEGL': No such file or directory
make[6]: *** [angle-build/libGLESv2.dll] Error 1
make[6]: *** Waiting for unfinished jobs....
cp: cannot create directory `angle-build/src/libGLESv2': File exists
make[6]: *** [angle-build/libEGL.dll] Error 1

Looks like Makefile error, not compiler. Any ideas?
Yeah, d3dx9.h I meant.  No idea about the compile error -- the build succeeds just fine locally, and even on the try server it built fine (other than the C compiler raising errors about missing headers and stuff).  Those errors are due to something else.

I'll remove the SDKs from my local LIB/INCLUDE and try to make it work here with adding them to the makefile, we'll see what happens.
Hrm, the patch that I wrote works just fine for me locally, with the SDK not being present in LIB/INCLUDE.  So I'm stumped, not sure why it doesn't find things on the tinderbox...
Note that for testing purposes, you can just cd into gfx/angle and run 'make angle-build/libGLESv2.dll' even without building any other prereqs to see if the part I'm trying to get going succeeds or not.
Is there a machine that has equivalent setup as the builders that I can just try things on directly?  Would be the fastest way to get things resolved, short of adding the SDK to LIB/INCLUDE in env.py.
d:\sdks\dx10sdk is in fact the wrong path; the right one is d:\sdks\dx10.  Running another test..
Rail, can I get an update on what's going on here?  I finally just added some 'ls' and 'cat' calls to the makefile and sent them to the try server:

ls: d:/sdks/dx10: No such file or directory
cat: d:/sdks/dx10/include/d3dx9.h: No such file or directory

1) Is this SDK installed in d:/sdks/dx10?
2) Is this SDK present on all try server machines?
SDK wasn't installed on the fast try slaves. I just requested package install for he rest of the slaves. It takes ~1 day to get the package deployed on all slaves usually. Please, try to run the build again tomorrow.

Sorry for the missed machines.
Success!  Now that things are actually present, build succeeded fine ;)

Can we get this deployed to all the build machines?
Good to hear this!
I requested package deploy for the rest of the build machines. I'll check if all of the machines got the package tomorrow.
I checked all builder slaves. All of them got the package except of these 3 slaves:

w32-ix-slave34.build -> bug 612288 (staging)
w32-ix-slave36.build -> bug 612288 (staging)
w32-ix-slave40.build -> bug 610301 (reboot requested)
Status: ASSIGNED → RESOLVED
Closed: 15 years ago → 14 years ago
Resolution: --- → FIXED
Did the paths get added to the LIB/INCLUDE env?
(In reply to comment #70)
> Did the paths get added to the LIB/INCLUDE env?

No.

IMHO, it would be better to add them using configure script (or mozilla-build), wouldn't it? In this case we'll get more portable solution.
No, because that would require everyone to have the directx sdk installed in the exact same location, which isn't the case... reopening until we get the env vars updated.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
IIRC, mozilla-biulds reads registry entries and finds locations of some tools (msvc for example). So it should be portable.

I'm fine with setting the env. BTW, to be sure, is there a possibility to break other branches by switching to the new SDK? Should we set the env only for particular branches?
Not that I can think of -- the only things that are there are directx headers/libraries, and we don't touch directx on other branches.  Ideally in the future we'll make mozilla-build find the dx sdk, but it's not necessary for now (and more complicated to do it).

For reference, I believe the install path will be in 'HKLM/Software/DirectX/Microsoft DirectX SDK (*)/InstallPath'  -- where * will be something like "February 2010", since you can have multiple SDKs installed.  So m-b will have to do some parsing to figure out which SDK to use.
Attached patch mozconfigs (obsolete) — — Splinter Review 
To be tested in staging. I tried to alter LIB/INCLUDE only for mozilla-central based branches.
Attachment #494116 - Flags: feedback?(bhearsum)
Comment on attachment 494116 [details] [diff] [review]
mozconfigs

Seems good to me. If this tests out fine, r+, too.
Attachment #494116 - Flags: feedback?(bhearsum) → feedback+
Unfortunately, setting env in mozconfigs didn't work. Somehow those variables are being ignored during compile step (while they appear in configure step).

Attached is a patch for buildbot-startup OPSI package which add required environment. Tested in staging.
Attachment #494116 - Attachment is obsolete: true
Attachment #496227 - Flags: review?(bhearsum)
Attachment #496227 - Flags: review?(bhearsum) → review+
The OPSI package is being deployed among the build slaves. I'll check its availability tomorrow.
All slaves have picked up the new packages.

Vlad, could you test the configuration in try? Something like http://hg.mozilla.org/try/rev/47d70938f0ed, but without setting LIB/INCLUDE.
Whiteboard: [buildslaves]
All done here. Feel free to reopen if something goes wrong.
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: