520765 - Win32 Fennec Desktop nightly builds are failing because of Win 7 SDK deployment

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

I am going to clobber as Nick did in bug 505504#c64 and see if it fixes it.

INCLUDE=d:\msvs9\VC\ATLMFC\INCLUDE;d:\msvs9\VC\INCLUDE;C:\Program Files\Microsoft SDKs\Windows\v6.0A\include;
LIB=d:\msvs9\VC\ATLMFC\LIB;d:\msvs9\VC\LIB;C:\Program Files\Microsoft SDKs\Windows\v6.0A\lib;
LIBPATH=C:\WINDOWS\Microsoft.NET\Framework\v3.5;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs9\VC\ATLMFC\LIB;d:\msvs9\VC\LIB;

checking for Windows SDK being recent enough... no
*** Fix above errors and then restart with               "make -f client.mk build"
make[2]: Leaving directory `/e/builds/moz2_slave/w32mob-1.9.2-nightly/mozilla-1.9.2'
make[1]: Leaving directory `/e/builds/moz2_slave/w32mob-1.9.2-nightly/mozilla-1.9.2'
configure: error: You are targeting Windows version 0x06010000, but your SDK only supports up to version 0x06000000. Install and use an updated SDK, or target a lower version using --with-windows-version. See https://developer.mozilla.org/En/Windows_SDK_versions for more details on fixing this.
make[2]: *** [configure] Error 1
make[1]: *** [/e/builds/moz2_slave/w32mob-1.9.2-nightly/mozilla-1.9.2/objdir/xulrunner/Makefile] Error 2
make: *** [build] Error 2

http://tinderbox.mozilla.org/showlog.cgi?log=Mobile/1254816002.1254816615.7998.gz&fulltext=1

http://tinderbox.mozilla.org/showlog.cgi?log=Mobile/1254816002.1254816839.10516.gz&fulltext=1

Comment 1

[[e-mail] (:bhearsum)]

LIB/INCLUDE seem to be set wrong, somehow.

Comment 2

[[e-mail] (:bhearsum)]

Looks like env.py is at fault again:
http://hg.mozilla.org/build/buildbotcustom/file/tip/env.py#l112 uses the 6.0 sdk in LIB/INCLUDE. Armen, can you try getting rid of the LIB/INCLUDE/SDKDIR variables in that dictionary?

Comment 3

[Armen [:armenzg]]

Attachment: remove environment variables for winmo win32 that override the desired new values

The Maemo desktop builds are running on staging

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 404824 [details] [diff] [review]
remove environment variables for winmo win32 that override the desired new values

changeset:   432:32d7e0c94360

Comment 5

[Armen [:armenzg]]

Masters updated and reconfigured.

Triggering new nightly builds.

Comment 6

[Armen [:armenzg]]

Green for both branches of win32 Fennec desktop builds

Resolving FIXED

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Did you consider the impact of these changes on WinMO and WinCE builds ? ie the important consumers of MozillaEnvironments['winmo-arm']

If I look at a WinCE build we now have 
 INCLUDE=d:\sdks\v7.0\\include;d:\sdks\v7.0\\include\atl;d:\msvs8\VC\ATLMFC\INCLUDE;d:\msvs8\VC\INCLUDE;d:\msvs8\VC\PlatformSDK\include;
  LIB=d:\sdks\v7.0\\lib;d:\msvs8\VC\ATLMFC\LIB;d:\msvs8\VC\LIB;d:\msvs8\VC\PlatformSDK\lib;d:\msvs8\SDK\v2.0\lib;;D:\mozilla-build\\atlthunk_compat
and PATH referring to msvs9. So the compiler executables are mismatched with the INCLUDE and LIB settings.

Comment 8

[Armen [:armenzg]]

No, I have not thought of them because I just did what I was suggested to do.

After your comments I looked at the code and I can see now that MozillaEnvironments['winmo-arm'] is shared between WinMo, WinCe and Win32 Desktop builds.

Shall I create a new section for Win32 Desktop builds?
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla2/mobile_config.py#96
Shall I point to some other environments section?
Does it need any environments variables or could it just work with the default values?

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

AIUI we need to use MSVC9 for WinCE and WinMO builds targeting the ARM architecture, while desktop builds of Fennec (ie XULRunner + the XUL app coming out of mobile-browser repo) could be the MSVC8 we use normally. So the desktop build would stop using the winmo-arm environment (which really means MSVC9), and that env would get updated to specify the 7.0 SDK instead of 6.0A. I'm testing this out in staging.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Ok, so with 3.6b1 imminent I'm going to back out 
 http://hg.mozilla.org/build/buildbotcustom/rev/32d7e0c94360
and re-break the desktop builds for win32. I don't want to regress the WinCE build by stirring the SDK soup a la comment #7.

If we want to switch WinMo/WinCE to the 7.0 SDK then we should do this:
 http://pastebin.mozilla.org/676303
That diff is against b0d53559b7be in buildbotcustom, the revision before 32d7e0c94360 landed. blassey is out today so could not get approval to make this change. We need to sync up with Mobile in general on the 7.0 SDK change (it may be a no-op but it's all a bit tangled right now and hard to tell). I had green builds in staging with that change.

To switch the desktop build to MSVS8 (VS2005) I think we'd want something like
  http://pastebin.mozilla.org/676305
When I tried that in staging I hit bug 506020 at 
  http://mxr.mozilla.org/mozilla-central/source/build/win32/Makefile.in#69
Renaming /d/mozilla-build/vim/vim72/install.exe lead to errors where it tried to copied MSVS8 files from the MSVS9 directory, presumably because of
http://hg.mozilla.org/build/buildbot-configs/file/de9cc3bae3a9/mozilla2/win32/mobile-desktop/nightly/mozconfig#l6

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

http://hg.mozilla.org/build/buildbotcustom/rev/b55b599f5f14
pm and pm02 reconfig'd at 2016 PDT.

Removing dependency on 3.6b1 tracking bug.

Comment 12

[Armen [:armenzg]]

Nick thanks for backing this out.

I will work on this on the next two days depending how my other stuff goes.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #12)
Re-assigning back to Armen based on this comment.

Comment 14

[Armen [:armenzg]]

Attachment: [buildbotcustom] add new win32-fennec environment variables section

Comment 15

[Armen [:armenzg]]

Attachment: [buildbot-configs] select "win32-fennec" environment variables for Win32 Fennec Desktop builds

Comment 16

[Armen [:armenzg]]

I have spent all morning on this when I thought it would be not that difficult to choose the right environment variables. I diffed the environment vairables from production logs of when it used to build, when it wouldn't build and with the environment variables of the slaves.

I barely know what I am doing so I decided to go the safe way.

The last 2 attached patches get the job done by creating a new environment variables section. This section is pretty much as the previous section but removing the same variables as in the backed out patch (https://bugzilla.mozilla.org/attachment.cgi?id=404824).

If this is good enough, let's do it; otherwise somebody take this bug since I want to focus on the multi-locale build for Fennec which has higher priority than fixing this.

Comment 17

[Armen [:armenzg]]

I have done a successful run of the attached patches on staging.

Nick, can we have this fixed separately from the issue that WinCE and WinMo are using older SDKs?

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Desktop fennec builds for win32 are broken right now, but WinCE builds are working. Desktop fennec builds for linux/mac are fine.

mfinkle is going to have a look at the conflicting env.setups required and see if he can help figure it out.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

(re the review request for attachments 406460 and 406462) 
I'm not at all fond of adding another hard-to-maintain environment to fix the Fennec desktop builds for Windows. It just makes it worse the next time we need to tweak the vars.

mfinkle, do you know if the desktop build must be compiled with VS2008 ? If it's a XUL app then perhaps we could (maybe even should) be using the VS2005 like the xulrunner nightly builds. If I try that with our standard build environment (VS2005 & the Windows 7 SDK) then configure says
 configure: warning: DirectDraw ddraw.h header not found or it's missing DDLOCK_WAITNOTBUSY, disabling DirectDraw surface.  If you have an older SDK (such as the CE5 SDK), try copying in ddraw.lib and ddraw.h from the WM6 SDK.
Is that relevant to a desktop build ? Thought that was WinMo/CE builds. There are other issues mentioned in comment #10 but I doubt they're not hard to fix.

Comment 20

[Armen [:armenzg]]

Taking this again.

(In reply to comment #19)
> (re the review request for attachments 406460 and 406462) 
> I'm not at all fond of adding another hard-to-maintain environment to fix the
> Fennec desktop builds for Windows. It just makes it worse the next time we need
> to tweak the vars.
> 
I do agree but that was the only thing I had at hand for a quick fix if this bug was super important for any reasons. I will try to use other existent environment variables sections OR reduce the one I proposed to a really short number of lines.

Comment 21

[Armen [:armenzg]]

After playing for a little while with the values of "win32-ref-platform" I ended up realizing that all I need was the PATH from "winmo-arm".
If I look at the resulting log I get these values:
> DEVENVDIR=d:\msvs8\Common7\IDE
> INCLUDE=d:\sdks\v7.0\\include;d:\sdks\v7.0\\include\atl;d:\msvs8\VC\ATLMFC\INCLUDE;d:\msvs8\VC\INCLUDE;d:\msvs8\VC\PlatformSDK\include;
> LIB=d:\sdks\v7.0\\lib;d:\msvs8\VC\ATLMFC\LIB;d:\msvs8\VC\LIB;d:\msvs8\VC\PlatformSDK\lib;d:\msvs8\SDK\v2.0\lib;;D:\mozilla-build\\atlthunk_compat
> LIBPATH=C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;d:\msvs8\VC\ATLMFC\LIB
> SDKDIR=d:\sdks\v7.0\
> SDKVER=7
Are these good enough?


Aside of that, the file env.py could be quite simplified and make it easier to maintain if we use things like reusing values from the ref-platform environments by simply doing string substitutions. Maybe we could file a bug for improving this file.

MozillaEnvironments['win32-fennec'] = {
#     "DEVENVDIR":       MozillaEnvironments['win32-ref-platform']['DEVENVDIR'].replace('msvs8\VC','msvs9'),
#     "INCLUDE":         MozillaEnvironments['win32-ref-platform']['INCLUDE'].replace('v6.0','v7.0'),
#     "LIB":             MozillaEnvironments['win32-ref-platform']['LIB'].replace('v6.0','v7.0') + \
#                        ';d:\\msvs8\\SDK\\v2.0\\lib;D:\\mozilla-build\\atlthunk_compat',
#     "SDKDIR":          MozillaEnvironments['win32-ref-platform']['SDKDIR'].replace('v6.0','v7.0'),
#     "SDKVER":          '7',
     # Note that I am importing from winmo-arm instead of win32-ref-platform
     "PATH":            MozillaEnvironments['winmo-arm']['PATH'],
}

Comment 22

[Armen [:armenzg]]

Attachment: [buildbotcustom] add new win32-fennec environment variables section

Comment 23

[Armen [:armenzg]]

Attachment: [buildbot-configs] select "win32-fennec" environment variables for Win32 Fennec Desktop builds

Comment 24

[Armen [:armenzg]]

These patches have worked on staging but I am running final clean builds for both branches to make sure that it all works as expected.

Comment 25

[Armen [:armenzg]]

Green runs on a clean run.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 408653 [details] [diff] [review]
[buildbotcustom] add new win32-fennec environment variables section 

r- 

>diff --git a/env.py b/env.py
>@@ -43,36 +43,38 @@ MozillaEnvironments['win32-ref-platform'
>+    "LIBPATH": 'C:\\WINDOWS\\Microsoft.NET\\Framework\\v2.0.50727;' + \
>+               'd:\\msvs9\\VC\\ATLMFC\\LIB;d:\\msvs9\\VC\\LIB;',

This was a stray hunk.

>+MozillaEnvironments['win32-fennec'] = {
>+    "PATH": MozillaEnvironments['winmo-arm']['PATH'],

This means we end up with a msvs9 style path, with msvs8 everything else (from MozBuild launching).

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 406460 [details] [diff] [review]
[buildbotcustom] add new win32-fennec environment variables section

In the interests of making the pain stop, I'm going to un-obsolete this attachment and grant r+.

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 408654 [details] [diff] [review]
[buildbot-configs] select "win32-fennec" environment variables for Win32 Fennec Desktop builds

http://hg.mozilla.org/build/buildbot-configs/rev/c0068547f206

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 406460 [details] [diff] [review]
[buildbotcustom] add new win32-fennec environment variables section

http://hg.mozilla.org/build/buildbotcustom/rev/f49fd8b839ad

Comment 30

[Armen [:armenzg]]

Masters reconfigured and triggered some nightly builds.

Comment 31

[Armen [:armenzg]]

Nightly builds for both branches
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/fennec-1.0b5pre.en-US.win32.zip
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/fennec-1.0b5pre.en-US.win32.zip

Resolving FIXED