1364137 - Windows SDK directory not detected properly on 64-bit python

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Kelsey Gilbert [:jgilbert]]

The SDKs we need are (on 64-bit systems) are in 'Program Files (x86)', not 'Program Files'. When we're using 32-bit python, we get the 32-bit registry view, with different keys and values from the 64-bit one. Using 64-bit python, we see only the Win10 SDK in 'Program Files', but this doesn't contain the includes and libraries.

Comment 1

[Kelsey Gilbert [:jgilbert]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4f9c5f19b2af71e01a348be2b9f08964b03d2c1

Comment 2

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1364137 - Find the right registry keys for python64. -

Also switch the key we check from 'Windows Kits' to 'Microsoft SDKs\Windows'.

Review commit: https://reviewboard.mozilla.org/r/138460/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/138460/

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 8866872 [details]
Bug 1364137 - Find the right registry keys for python64. -

https://reviewboard.mozilla.org/r/138460/#review143934

::: build/moz.configure/windows.configure:40
(Diff revision 1)
>      if host.kernel != 'WINNT':
>          return ()
>  
> -    return tuple(x[1] for x in get_registry_values(
> -        r'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Kits\Installed Roots'
> -        r'\KitsRoot*'))
> +    software_key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, 'SOFTWARE')
> +    try:
> +        software_key = winreg.OpenKey(software_key, 'WOW6432Node')

Interesting, that's something I had done to find Visual Studio, but not the SDK.

However, I'd rather keep the get_registry_values function, even if it's only used here. Because if we end up needing it more in the future (we only recently removed other uses), I don't want ad-hoc code using winreg being written.

The status of HKLM\SOFTWARE vs HKLM\SOFTWARE\Wow6432Node entries is unclear. I remember that for Visual Studio, I had the opposite problem you have, which is that the Wow6432Node entries didn't have everything.

So it seems we should query both in all cases, and aggregate from there, picking the right things that exist.

::: build/moz.configure/windows.configure:45
(Diff revision 1)
> -        r'\KitsRoot*'))
> +        software_key = winreg.OpenKey(software_key, 'WOW6432Node')
> +    except WindowsError:
> +        # 32-bit python
> +        pass
> +
> +    sdks_key = winreg.OpenKey(software_key, 'Microsoft\Microsoft SDKs\Windows')

I have both SDK 8.1 and 10.0 installed, and only 8.1 is represented under MS\MS SDKs\Windows. They are however both under MS\Windows Kits\Installed Roots\KitsRoot* (in both Wow6432Node and normal).

Comment 4

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8866872 [details]
Bug 1364137 - Find the right registry keys for python64. -

https://reviewboard.mozilla.org/r/138460/#review143934

> Interesting, that's something I had done to find Visual Studio, but not the SDK.
> 
> However, I'd rather keep the get_registry_values function, even if it's only used here. Because if we end up needing it more in the future (we only recently removed other uses), I don't want ad-hoc code using winreg being written.
> 
> The status of HKLM\SOFTWARE vs HKLM\SOFTWARE\Wow6432Node entries is unclear. I remember that for Visual Studio, I had the opposite problem you have, which is that the Wow6432Node entries didn't have everything.
> 
> So it seems we should query both in all cases, and aggregate from there, picking the right things that exist.

I really don't like the get_registry_values function because it's hard to figure out how to use, in comparison to the ample documentation for winreg. It was easier for me to understand how to use winreg than to understand what I could or couldn't expect from get_registry_values.

If a node is not in Wow6432Node, it won't show up for 32-bit programs, including our current python32 system. I would prefer not to cargo-cult and check-all-the-things.

> I have both SDK 8.1 and 10.0 installed, and only 8.1 is represented under MS\MS SDKs\Windows. They are however both under MS\Windows Kits\Installed Roots\KitsRoot* (in both Wow6432Node and normal).

Is it a usable 10, and is it installed properly? If you re-install it, does it still not show in \"Microsoft SDKs"\Windows?
Specifically, does your 10 SDK have the actual include/libraries paths? I had at least one 10 SDK that only had useless (to me) items, no includes or library files.

Comment 5

[Mike Hommey [:glandium]]

(In reply to Jeff Gilbert [:jgilbert] from comment #4)
> Comment on attachment 8866872 [details]
> Bug 1364137 - Find the right registry keys for python64. -
> 
> https://reviewboard.mozilla.org/r/138460/#review143934
> 
> > Interesting, that's something I had done to find Visual Studio, but not the SDK.
> > 
> > However, I'd rather keep the get_registry_values function, even if it's only used here. Because if we end up needing it more in the future (we only recently removed other uses), I don't want ad-hoc code using winreg being written.
> > 
> > The status of HKLM\SOFTWARE vs HKLM\SOFTWARE\Wow6432Node entries is unclear. I remember that for Visual Studio, I had the opposite problem you have, which is that the Wow6432Node entries didn't have everything.
> > 
> > So it seems we should query both in all cases, and aggregate from there, picking the right things that exist.
> 
> I really don't like the get_registry_values function because it's hard to
> figure out how to use, in comparison to the ample documentation for winreg.
> It was easier for me to understand how to use winreg than to understand what
> I could or couldn't expect from get_registry_values.

If something is not clear in the get_registry_values documentation, let's fix that. Your use case is pretty straightforward:
get_registry_values(r'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v*\InstallationFolder') will return something like:
 (
    ('vx.y', 'path'),
    ...
 )

> If a node is not in Wow6432Node, it won't show up for 32-bit programs,
> including our current python32 system. I would prefer not to cargo-cult and
> check-all-the-things.

If a node is not in HKLM\SOFTWARE\Wow6432Node in the 64-bits view, it won't show up under HKLM\SOFTWARE, true. But the point is that HKLM\SOFTWARE in the 64-bits view may contain values, and HKLM\SOFTWARE\Wow6432Node others, and the ones that work for the build can be in either place.

> > I have both SDK 8.1 and 10.0 installed, and only 8.1 is represented under MS\MS SDKs\Windows. They are however both under MS\Windows Kits\Installed Roots\KitsRoot* (in both Wow6432Node and normal).
> 
> Is it a usable 10, and is it installed properly? If you re-install it, does
> it still not show in \"Microsoft SDKs"\Windows?
> Specifically, does your 10 SDK have the actual include/libraries paths? I
> had at least one 10 SDK that only had useless (to me) items, no includes or
> library files.

It's the SDK that came installed alongside MSVC iirc, and it has the includes and library files. In fact, that's the SDK the build system picks to build Firefox in a start-shell.bat shell on that machine.

Comment 6

[Mike Hommey [:glandium]]

So, looking more closely, the SDK is *not* complete. It only contains the UCRT, which is required to build. And the build system needs to find both 8.1 and 10 SDK paths in that case, which using the Microsoft SDKs keys would break.

Comment 7

[Bob Owen (:bobowen)]

I couldn't quite reproduce this issue, I was using a fresh Win7 VM with VS2017 and a version of MozillaBuild 3.0 (from Bug 1346656).

While the Win8.1 SDK in the 64-bit registry view pointed to |Program Files| the Win10 SDK pointed to |Program Files (x86)|, so things worked.

However it seems that different people's installations can be fairly inconsistent with these different registry views, so here's a try run with a patch to get both lots, using windows own access mask functionality:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b65d444f995a5c4e2b52ec4d973c38ce89206e3d

Comment 8

[Bob Owen (:bobowen)]

Attachment: Get both 32-bit and 64-bit registry values when searching for the Windows SDK

The values that we need to find in the registry can be inconsistent across
different installations, so we retrieve values from both views in our search
for a valid SDK. This also ensures this works for 32-bit and 64-bit python.

Comment 9

[Mike Hommey [:glandium]]

Comment on attachment 8887564 [details] [diff] [review]
Get both 32-bit and 64-bit registry values when searching for the Windows SDK

Review of attachment 8887564 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/moz.configure/util.configure
@@ +240,5 @@
>  #   yields e.g.:
>  #     (r'C:\Program Files (x86)\Windows Kits\8.1\',)
>  #
>  #   get_registry_values(r'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\'
> +#                       r'Windows Kits\Installed Roots\KitsRoot8.1', True)

It would be preferable to pass get_32_and_64_bit=True because True alone doesn't tell much.

::: build/moz.configure/windows.configure
@@ +37,2 @@
>          r'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Kits\Installed Roots'
> +        r'\KitsRoot*', True))

same here.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2626221a97cc
Get both 32-bit and 64-bit registry values when searching for the Windows SDK. r=glandium

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/2626221a97cc

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/c2635cb28a2f