1522931 - OSX SDK version check doesn't work when using default SDK

Status: REOPENED

(Firefox Build System :: General, enhancement)

Description

[Michael Shal [:mshal]]

Bug 1500504 added a version check for the SDK, but that only applies if --with-macos-sdk is used. If that option is not present, the default SDK is used and we don't check its version.

Comment 1

[Michael Shal [:mshal]]

Attachment: Bug 1522931 - always check the OSX SDK version; r?#firefox-build-system-reviewers

Bug 1500504 added a version check for the SDK, but it only does the
check if --with-macos-sdk is used. We should also check the version when
using the default SDK.

Note that this means we now set MACOS_SDK_DIR to be the default SDK even
if it wasn't set explicitly from --with-macos-sdk

Comment 2

[Michael Shal [:mshal]]

Comment on attachment 9039297 [details]
Bug 1522931 - always check the OSX SDK version; r?#firefox-build-system-reviewers

:bholley, can you verify that this works for you? I'm curious if it is able to parse the SDKSettings.plist file that you posted.

Comment 3

[Bobby Holley (:bholley)]

Attachment: output

I get the following output:

Comment 4

[Michael Shal [:mshal]]

Sounds like we'll need a different solution here, since the SDKSettings.plist file format is not parseable in 10.14+ with plistlib in python. Anyone have ideas? I'm not sure what else to do here, since this needs to work on both local mac builds and cross-linux builds.

Comment 5

[Brendan Dahl [:bdahl]]

Attachment: SDKSettings.plist

As requested by glandium.

Comment 6

[Mike Hommey [:glandium]]

So the problem is that the plist in the 10.14 SDK is binary, and plistlib doesn't support that format until python 3.4.

The biplist library on pypi has essentially the same API as plistlib, and can be used to read binary plists. We could vendor it and use it. Its readPlist even falls back to plistlib's when it's used on a non-binary plist.

Mike, do you want to update your patch?

Comment 7

[Michael Shal [:mshal]]

(In reply to Mike Hommey [:glandium] from comment #6)

Mike, do you want to update your patch?

Sure thing, patches incoming...

Comment 8

[Michael Shal [:mshal]]

Attachment: Bug 1522931 - Vendor biplist; r=froydnj

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/71191e5499a1
Vendor biplist; r=froydnj

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/27214432fe48
always check the OSX SDK version; r=firefox-build-system-reviewers,chmanchester

Comment 11

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/71191e5499a1
https://hg.mozilla.org/mozilla-central/rev/27214432fe48

Comment 12

[Natalia Csoregi [:nataliaCs]]

Backed out for breaking macOS build on 10.14.
backout: https://hg.mozilla.org/mozilla-central/rev/43c4699b1770e430e57a1b77604d7bbb36936b6b

Comment 13

[Miko Mynttinen]

This is breaking my local build on Mojave, with 10.13 SDK.

 0:03.13 Adding configure options from /Users/miko/.mozconfigs/mac-release
 0:03.13   --enable-release
 0:03.13   --enable-optimize
 0:03.13   --enable-debug-symbols
 0:03.13   --enable-profiling
 0:03.13   --enable-instruments
 0:03.14   --disable-crashreporter
 0:03.14   --with-macos-sdk=/Library/Developer/SDKs/MacOSX10.13.sdk
[...]
 0:09.65 checking whether the C compiler (/usr/bin/clang -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu99   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) works... (cached) yes
 0:09.65 checking whether the C compiler (/usr/bin/clang -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu99   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) is a cross-compiler... no
 0:09.65 checking whether we are using GNU C... (cached) yes
 0:09.65 checking whether /usr/bin/clang -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu99 accepts -g... (cached) yes
 0:09.65 checking for c++... (cached) /usr/bin/clang++ -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu++14
 0:09.65 checking whether the C++ compiler (/usr/bin/clang++ -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu++14   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) works... (cached) yes
 0:09.65 checking whether the C++ compiler (/usr/bin/clang++ -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu++14   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) is a cross-compiler... no
[...]
 0:15.84 js/src> running /Users/miko/Code/mozilla-unified/configure.py --enable-project=js --disable-crashreporter --host=x86_64-apple-darwin18.5.0 --target=x86_64-apple-darwin18.5.0 --enable-tests --disable-debug --disable-rust-debug --enable-release --enable-optimize --enable-xcode-checks --without-ccache --without-toolchain-prefix --enable-debug-symbols --disable-profile-generate --disable-profile-use --without-pgo-profile-path --disable-lto --disable-address-sanitizer --disable-undefined-sanitizer --disable-coverage --disable-linker --disable-clang-plugin --disable-mozsearch-plugin --disable-stdcxx-compat --enable-jemalloc --enable-replace-malloc --without-linux-headers --disable-warnings-as-errors --disable-valgrind --without-libclang-path --without-clang-path --disable-js-shell --disable-shared-js --disable-export-js --enable-ion --disable-simulator --enable-instruments --disable-callgrind --enable-profiling --disable-vtune --disable-gc-trace --disable-gczeal --disable-small-chunk-size --disable-trace-logging --disable-oom-breakpoint --disable-perf --disable-jitspew --disable-masm-verbose --disable-more-deterministic --enable-ctypes --without-system-ffi --disable-fuzzing --disable-pipeline-operator --enable-binast --enable-cranelift --disable-wasm-codegen-debug --enable-typed-objects --enable-wasm-bulk-memory --enable-wasm-reftypes --enable-wasm-gc --enable-wasm-private-reftypes --with-nspr-cflags=-I/Users/miko/Code/mozilla-unified/obj-ff-release/dist/include/nspr --with-nspr-libs=-L/Users/miko/Code/mozilla-unified/obj-ff-release/dist/lib -lnspr4 -lplc4 -lplds4 --prefix=/Users/miko/Code/mozilla-unified/obj-ff-release/dist JS_STANDALONE=
 0:15.86 js/src> checking for vcs source checkout... hg
 0:15.89 js/src> checking for a shell... /bin/sh
 0:15.90 js/src> checking for host system type... x86_64-apple-darwin18.5.0
 0:15.92 js/src> checking for target system type... x86_64-apple-darwin18.5.0
 0:16.22 js/src> checking whether cross compiling... no
 0:16.25 js/src> checking for Python 3... /usr/local/bin/python3 (3.7.3)
 0:16.25 js/src> checking for hg... /usr/local/bin/hg
 0:16.41 js/src> checking for Mercurial version... 4.9.1
 0:16.59 js/src> checking for sparse checkout... no
 0:16.60 js/src> checking for yasm... /usr/local/bin/yasm
 0:16.61 js/src> checking yasm version... 1.3.0
 0:16.62 js/src> ERROR: SDK version "10.14" is unsupported. Please downgrade to version 10.13. You may need to point to it using --with-macos-sdk=<path> in your mozconfig. Various SDK versions are available from https://github.com/phracker/MacOSX-SDKs
 0:16.68 *** Fix above errors and then restart with               "./mach build"

I am guessing the problem is that SDK setting is not properly propagated to JS configure.

Comment 14

[Michael Shal [:mshal]]

Looks like you are correct that the flag wasn't getting passed to js' configure. Can you verify that the latest patches work for you :miko? I was able to reproduce it by hacking some SDKSettings.plist files locally, but I wanted to make sure it works in a real setting.

Comment 15

[Mike Hommey [:glandium]]

That's because --with-macos-sdk is not a js_option.

Comment 16

[Miko Mynttinen]

(In reply to Michael Shal [:mshal] from comment #14)

Looks like you are correct that the flag wasn't getting passed to js' configure. Can you verify that the latest patches work for you :miko? I was able to reproduce it by hacking some SDKSettings.plist files locally, but I wanted to make sure it works in a real setting.

I applied the patches locally over the latest m-c, and the build worked.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/12e5e835a768
Vendor biplist; r=froydnj
https://hg.mozilla.org/integration/autoland/rev/3d0236f985f8
always check the OSX SDK version; r=firefox-build-system-reviewers,chmanchester

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/12e5e835a768
https://hg.mozilla.org/mozilla-central/rev/3d0236f985f8

Comment 19

[Kartikaya Gupta (email:[e-mail])]

I'm on macOS 10.13.6 but apparently I have the 10.14 SDK, and that was building fine until this landed. Now it fails to build because it's all like "your SDK is too new". Which I guess is the intended behaviour, but if it was working fine before, does that mean we really do support the 10.14 SDK?

kats@kgupta-mbp gecko-mac$ xcrun --sdk macosx --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
kats@kgupta-mbp gecko-mac$ ls -al /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/
total 0
drwxr-xr-x  4 root  wheel  128 31 Oct 09:06 .
drwxr-xr-x  5 root  wheel  160 19 Oct 21:18 ..
drwxr-xr-x  7 root  wheel  224 31 Oct 09:05 MacOSX.sdk
lrwxr-xr-x  1 root  wheel   10 26 Sep  2018 MacOSX10.14.sdk -> MacOSX.sdk

It looks like I've had the 10.14 SDK for a while, if the timestamps are to be believed. I certainly haven't installed any macOS updates in the past few days (I keep deferring the upgrade to 10.14).

Comment 20

[Jonathan Kew [:jfkthame]]

This seems to have broken my local build. :(

Scenario: I'm running macOS 10.13.6. I've been building with the SDK that came with my version of Xcode (10.1), which appears to be SDK 10.14. This has been working fine. (Maybe such builds would give problems if running on 10.14? For my local development, however, there hasn't been any issue.)

After updating to latest m-c, the build errors out, telling me SDK 10.14 is not supported and I should downgrade to SDK 10.13:

 0:05.37 ERROR: SDK version "10.14" is unsupported. Please downgrade to version 10.13. You may need to point to it using --with-macos-sdk=<path> in your mozconfig. Various SDK versions are available from https://github.com/phracker/MacOSX-SDKs

OK, I grabbed SDK 10.13 from https://github.com/phracker/MacOSX-SDKs as recommended by the error message, and added the appropriate --with-macos-sdk to a mozconfig.

So now configure is happy; but my build fails:

 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/xpcom/io/nsLocalFileUnix.cpp:45:
 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/xpcom/io/nsLocalFile.h:44:
 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/xpcom/io/nsLocalFileUnix.h:24:
 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/obj-x86_64-apple-darwin17.7.0/dist/include/nsILocalFileMac.h:17:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:23:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:87:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/Components.h:26:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/Files.h:56:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/DiskArbitration.framework/Headers/DADisk.h:28:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/IOKit.framework/Headers/IOKitLib.h:49:
 7:27.22 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/IOKit.framework/Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 7:27.22 #include <IOKit/IOReturn.h>
 7:27.22          ^~~~~~~~~~~~~~~~~~
 7:28.17 BUILDSTATUS BUILD_VERBOSE xpcom/base
 7:28.19 1 error generated.
 7:28.21 make[4]: *** [nsLocalFileUnix.o] Error 1

Any idea what's wrong?

Comment 21

[Kartikaya Gupta (email:[e-mail])]

I'm trying a build now with sdk_max_version bumped to 10.14, I expect that to give me the same results as I used to get before this landed. Will confirm when it's done.

Comment 22

[Jonathan Kew [:jfkthame]]

Yes, that works, but it's a bit annoying to have to use a local build system patch on what ought to be a pretty straightforward development configuration.

Comment 23

[Kartikaya Gupta (email:[e-mail])]

My build with that patch failed:

31:51.17 In file included from /Users/kats/zspace/gecko-mac/obj-host-opt/widget/cocoa/Unified_mm_widget_cocoa0.mm:11:
31:51.17 /Users/kats/zspace/gecko-mac/widget/cocoa/GfxInfo.mm:78:34: warning: 'CGDisplayIOServicePort' is deprecated: first deprecated in macOS 10.9 - No longer supported [-Wdeprecated-declarations]
31:51.17   io_registry_entry_t dsp_port = CGDisplayIOServicePort(kCGDirectMainDisplay);
31:51.17                                  ^
31:51.18 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreGraphics.framework/Headers/CGDisplayConfiguration.h:372:24: note: 'CGDisplayIOServicePort' has been explicitly marked deprecated here
31:51.18 CG_EXTERN io_service_t CGDisplayIOServicePort(CGDirectDisplayID display)
31:51.18                        ^
31:51.18 In file included from /Users/kats/zspace/gecko-mac/obj-host-opt/widget/cocoa/Unified_mm_widget_cocoa0.mm:110:
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsAppShell.mm:316:15: warning: 'loadNibFile:externalNameTable:withZone:' is deprecated: first deprecated in macOS 10.8 [-Wdeprecated-declarations]
31:51.19     [NSBundle loadNibFile:[NSString stringWithUTF8String:(const char*)nibPath.get()]
31:51.19               ^
31:51.19 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/AppKit.framework/Headers/NSNibLoading.h:30:1: note: 'loadNibFile:externalNameTable:withZone:' has been explicitly marked deprecated here
31:51.19 + (BOOL)loadNibFile:(null_unspecified NSString *)fileName externalNameTable:(null_unspecified NSDictionary *)context withZone:(null_unspecified NSZone *)zone NS_DEPRECATED_MAC(10_0, 10_8);
31:51.19 ^
31:51.19 In file included from /Users/kats/zspace/gecko-mac/obj-host-opt/widget/cocoa/Unified_mm_widget_cocoa0.mm:137:
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1114:24: error: 'authorizationStatusForMediaType:' is only available on macOS 10.14 or newer [-Werror,-Wunguarded-availability-new]
31:51.19       [AVCaptureDevice authorizationStatusForMediaType:aMediaType]);
31:51.19                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/AVFoundation.framework/Headers/AVCaptureDevice.h:1479:1: note: 'authorizationStatusForMediaType:' has been explicitly marked partial here
31:51.19 + (AVAuthorizationStatus)authorizationStatusForMediaType:(AVMediaType)mediaType API_AVAILABLE(macos(10.14), ios(7.0));
31:51.19 ^
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1114:24: note: enclose 'authorizationStatusForMediaType:' in an @available check to silence this warning
31:51.19       [AVCaptureDevice authorizationStatusForMediaType:aMediaType]);
31:51.19                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1209:20: error: 'requestAccessForMediaType:completionHandler:' is only available on macOS 10.14 or newer [-Werror,-Wunguarded-availability-new]
31:51.19   [AVCaptureDevice requestAccessForMediaType:aType completionHandler:aHandler];
31:51.19                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/AVFoundation.framework/Headers/AVCaptureDevice.h:1502:1: note: 'requestAccessForMediaType:completionHandler:' has been explicitly marked partial here
31:51.19 + (void)requestAccessForMediaType:(AVMediaType)mediaType completionHandler:(void (^)(BOOL granted))handler API_AVAILABLE(macos(10.14), ios(7.0));
31:51.19 ^
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1209:20: note: enclose 'requestAccessForMediaType:completionHandler:' in an @available check to silence this warning
31:51.19   [AVCaptureDevice requestAccessForMediaType:aType completionHandler:aHandler];
31:51.19                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 2 warnings and 2 errors generated.
31:51.19 make[4]: *** [Unified_mm_widget_cocoa0.o] Error 1
31:51.19 make[4]: *** Waiting for unfinished jobs....
31:51.19 editor/txmgr/tests
31:51.58 editor/composer
31:53.60 layout/style
31:57.70 layout/style/test/gtest
31:59.11 make[3]: *** [widget/cocoa/target] Error 2
31:59.11 make[3]: *** Waiting for unfinished jobs....
32:11.56 make[2]: *** [compile] Error 2
32:11.56 make[1]: *** [default] Error 2
32:11.56 make: *** [build] Error 2

Comment 24

[Kartikaya Gupta (email:[e-mail])]

I filed bug 1544418 to track this; let's continue discussion over there.

Comment 25

[Michael Shal [:mshal]]

This was backed out by bug 1544418 - :spohl, I could use some guidance on what the current set of acceptable SDKs for local developers is, and maybe some help on why adding the check results in failures on otherwise good versions. (eg: https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c17 and https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c18)

Comment 26

[Stephen A Pohl [:spohl]]

(In reply to Michael Shal [:mshal] from comment #25)

This was backed out by bug 1544418 - :spohl, I could use some guidance on what the current set of acceptable SDKs for local developers is, and maybe some help on why adding the check results in failures on otherwise good versions. (eg: https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c17 and https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c18)

The only SDK that we officially support at the moment is the 10.11 SDK. I will not have the bandwidth to test various SDK/macOS combinations. If a developer encounters issues with an SDK other than 10.11 (for example visual glitches), the first step should be to rebuild with the 10.11 SDK.

If building with the 10.14 SDK is now working and no longer results in completely black windows we can leave this SDK check removed. We should be ready to reintroduce a similar check if a future SDK were to break things again.

Comment 27

[Bobby Holley (:bholley)]

To anyone else who's a bit concerned about the unverified nature of the phracker SDK repository:

You can download an official copy of the 10.11 SDK by downloading Xcode 7.1.1 from apple's developer portal [1] [2]. Mount the DMG, right-click on the installer, go to "show package contents". Then go Contents->Developer->Platforms->MacOSX.platform->Developer->SDKs to find MacOSX10.11.sdk.

I have verified that this very-nearly matches this 10.11 SDK on phracker: [3] [4] (verify the checksum!). The differences between the two are available for perusal at [5], and do not appear to be malicious.

[1] https://download.developer.apple.com/Developer_Tools/Xcode_7.1.1/Xcode_7.1.1.dmg
[2] SHA1: 33163b3cbff846224a2661c08ef8e737f0da0762
[3] https://github.com/phracker/MacOSX-SDKs/releases/download/MacOSX10.11.sdk/MacOSX10.11.sdk.tar.xz
[4] SHA1: f3430e3d923644e66c0c13f7a48754e7b6aa2e3f
[5] https://gist.github.com/bholley/4ba2e619912df44d97be7d8634921e20

Comment 28

[Mark Banner (:standard8)]

Given Mike Hommey's comment in dev.platform that pretty much any SDK should work properly now, I'm removing this from mach-busted. It is unclear if it still remains an issue or not, so leaving open for now.

https://groups.google.com/g/mozilla.dev.platform/c/9wwJJy5HCfM/m/SXlr21h_AAAJ