OSX SDK version check doesn't work when using default SDK

REOPENED
Unassigned

Status

enhancement
REOPENED
4 months ago
14 days ago

People

(Reporter: mshal, Unassigned)

Tracking

(Blocks 1 bug, Regressed 1 bug)

unspecified
mozilla68
Dependency tree / graph

Firefox Tracking Flags

(firefox68 affected)

Details

Attachments

(4 attachments)

Reporter

Description

4 months ago

Bug 1500504 added a version check for the SDK, but that only applies if --with-macos-sdk is used. If that option is not present, the default SDK is used and we don't check its version.

Reporter

Comment 1

4 months ago

Bug 1500504 added a version check for the SDK, but it only does the
check if --with-macos-sdk is used. We should also check the version when
using the default SDK.

Note that this means we now set MACOS_SDK_DIR to be the default SDK even
if it wasn't set explicitly from --with-macos-sdk

Reporter

Comment 2

4 months ago

Comment on attachment 9039297 [details]
Bug 1522931 - always check the OSX SDK version; r?#firefox-build-system-reviewers

:bholley, can you verify that this works for you? I'm curious if it is able to parse the SDKSettings.plist file that you posted.

Attachment #9039297 - Flags: feedback?(bobbyholley)
Posted file output

I get the following output:

Attachment #9039297 - Flags: feedback?(bobbyholley)
Reporter

Comment 4

4 months ago

Sounds like we'll need a different solution here, since the SDKSettings.plist file format is not parseable in 10.14+ with plistlib in python. Anyone have ideas? I'm not sure what else to do here, since this needs to work on both local mac builds and cross-linux builds.

Posted file SDKSettings.plist

As requested by glandium.

Attachment #9055663 - Attachment mime type: application/octet-stream → text/plain

So the problem is that the plist in the 10.14 SDK is binary, and plistlib doesn't support that format until python 3.4.

The biplist library on pypi has essentially the same API as plistlib, and can be used to read binary plists. We could vendor it and use it. Its readPlist even falls back to plistlib's when it's used on a non-binary plist.

Mike, do you want to update your patch?

Flags: needinfo?(mshal)
Reporter

Comment 7

2 months ago

(In reply to Mike Hommey [:glandium] from comment #6)

Mike, do you want to update your patch?

Sure thing, patches incoming...

Flags: needinfo?(mshal)

Comment 10

a month ago
Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/27214432fe48
always check the OSX SDK version; r=firefox-build-system-reviewers,chmanchester

Comment 11

a month ago
bugherder
Status: NEW → RESOLVED
Last Resolved: a month ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68

Backed out for breaking macOS build on 10.14.
backout: https://hg.mozilla.org/mozilla-central/rev/43c4699b1770e430e57a1b77604d7bbb36936b6b

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: mozilla68 → ---

This is breaking my local build on Mojave, with 10.13 SDK.

 0:03.13 Adding configure options from /Users/miko/.mozconfigs/mac-release
 0:03.13   --enable-release
 0:03.13   --enable-optimize
 0:03.13   --enable-debug-symbols
 0:03.13   --enable-profiling
 0:03.13   --enable-instruments
 0:03.14   --disable-crashreporter
 0:03.14   --with-macos-sdk=/Library/Developer/SDKs/MacOSX10.13.sdk
[...]
 0:09.65 checking whether the C compiler (/usr/bin/clang -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu99   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) works... (cached) yes
 0:09.65 checking whether the C compiler (/usr/bin/clang -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu99   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) is a cross-compiler... no
 0:09.65 checking whether we are using GNU C... (cached) yes
 0:09.65 checking whether /usr/bin/clang -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu99 accepts -g... (cached) yes
 0:09.65 checking for c++... (cached) /usr/bin/clang++ -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu++14
 0:09.65 checking whether the C++ compiler (/usr/bin/clang++ -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu++14   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) works... (cached) yes
 0:09.65 checking whether the C++ compiler (/usr/bin/clang++ -isysroot /Library/Developer/SDKs/MacOSX10.13.sdk -std=gnu++14   -Wl,-syslibroot,/Library/Developer/SDKs/MacOSX10.13.sdk) is a cross-compiler... no
[...]
 0:15.84 js/src> running /Users/miko/Code/mozilla-unified/configure.py --enable-project=js --disable-crashreporter --host=x86_64-apple-darwin18.5.0 --target=x86_64-apple-darwin18.5.0 --enable-tests --disable-debug --disable-rust-debug --enable-release --enable-optimize --enable-xcode-checks --without-ccache --without-toolchain-prefix --enable-debug-symbols --disable-profile-generate --disable-profile-use --without-pgo-profile-path --disable-lto --disable-address-sanitizer --disable-undefined-sanitizer --disable-coverage --disable-linker --disable-clang-plugin --disable-mozsearch-plugin --disable-stdcxx-compat --enable-jemalloc --enable-replace-malloc --without-linux-headers --disable-warnings-as-errors --disable-valgrind --without-libclang-path --without-clang-path --disable-js-shell --disable-shared-js --disable-export-js --enable-ion --disable-simulator --enable-instruments --disable-callgrind --enable-profiling --disable-vtune --disable-gc-trace --disable-gczeal --disable-small-chunk-size --disable-trace-logging --disable-oom-breakpoint --disable-perf --disable-jitspew --disable-masm-verbose --disable-more-deterministic --enable-ctypes --without-system-ffi --disable-fuzzing --disable-pipeline-operator --enable-binast --enable-cranelift --disable-wasm-codegen-debug --enable-typed-objects --enable-wasm-bulk-memory --enable-wasm-reftypes --enable-wasm-gc --enable-wasm-private-reftypes --with-nspr-cflags=-I/Users/miko/Code/mozilla-unified/obj-ff-release/dist/include/nspr --with-nspr-libs=-L/Users/miko/Code/mozilla-unified/obj-ff-release/dist/lib -lnspr4 -lplc4 -lplds4 --prefix=/Users/miko/Code/mozilla-unified/obj-ff-release/dist JS_STANDALONE=
 0:15.86 js/src> checking for vcs source checkout... hg
 0:15.89 js/src> checking for a shell... /bin/sh
 0:15.90 js/src> checking for host system type... x86_64-apple-darwin18.5.0
 0:15.92 js/src> checking for target system type... x86_64-apple-darwin18.5.0
 0:16.22 js/src> checking whether cross compiling... no
 0:16.25 js/src> checking for Python 3... /usr/local/bin/python3 (3.7.3)
 0:16.25 js/src> checking for hg... /usr/local/bin/hg
 0:16.41 js/src> checking for Mercurial version... 4.9.1
 0:16.59 js/src> checking for sparse checkout... no
 0:16.60 js/src> checking for yasm... /usr/local/bin/yasm
 0:16.61 js/src> checking yasm version... 1.3.0
 0:16.62 js/src> ERROR: SDK version "10.14" is unsupported. Please downgrade to version 10.13. You may need to point to it using --with-macos-sdk=<path> in your mozconfig. Various SDK versions are available from https://github.com/phracker/MacOSX-SDKs
 0:16.68 *** Fix above errors and then restart with               "./mach build"

I am guessing the problem is that SDK setting is not properly propagated to JS configure.

Attachment #9056266 - Attachment description: Bug 1522931 - Vendor biplist; r?#firefox-build-system-reviewers → Bug 1522931 - Vendor biplist; r=froydnj
Reporter

Comment 14

a month ago

Looks like you are correct that the flag wasn't getting passed to js' configure. Can you verify that the latest patches work for you :miko? I was able to reproduce it by hacking some SDKSettings.plist files locally, but I wanted to make sure it works in a real setting.

Flags: needinfo?(mikokm)

That's because --with-macos-sdk is not a js_option.

(In reply to Michael Shal [:mshal] from comment #14)

Looks like you are correct that the flag wasn't getting passed to js' configure. Can you verify that the latest patches work for you :miko? I was able to reproduce it by hacking some SDKSettings.plist files locally, but I wanted to make sure it works in a real setting.

I applied the patches locally over the latest m-c, and the build worked.

Flags: needinfo?(mikokm)

Comment 17

a month ago
Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/12e5e835a768
Vendor biplist; r=froydnj
https://hg.mozilla.org/integration/autoland/rev/3d0236f985f8
always check the OSX SDK version; r=firefox-build-system-reviewers,chmanchester

Comment 18

a month ago
bugherder
Status: REOPENED → RESOLVED
Last Resolved: a month agoa month ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68

I'm on macOS 10.13.6 but apparently I have the 10.14 SDK, and that was building fine until this landed. Now it fails to build because it's all like "your SDK is too new". Which I guess is the intended behaviour, but if it was working fine before, does that mean we really do support the 10.14 SDK?

kats@kgupta-mbp gecko-mac$ xcrun --sdk macosx --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
kats@kgupta-mbp gecko-mac$ ls -al /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/
total 0
drwxr-xr-x  4 root  wheel  128 31 Oct 09:06 .
drwxr-xr-x  5 root  wheel  160 19 Oct 21:18 ..
drwxr-xr-x  7 root  wheel  224 31 Oct 09:05 MacOSX.sdk
lrwxr-xr-x  1 root  wheel   10 26 Sep  2018 MacOSX10.14.sdk -> MacOSX.sdk

It looks like I've had the 10.14 SDK for a while, if the timestamps are to be believed. I certainly haven't installed any macOS updates in the past few days (I keep deferring the upgrade to 10.14).

This seems to have broken my local build. :(

Scenario: I'm running macOS 10.13.6. I've been building with the SDK that came with my version of Xcode (10.1), which appears to be SDK 10.14. This has been working fine. (Maybe such builds would give problems if running on 10.14? For my local development, however, there hasn't been any issue.)

After updating to latest m-c, the build errors out, telling me SDK 10.14 is not supported and I should downgrade to SDK 10.13:

 0:05.37 ERROR: SDK version "10.14" is unsupported. Please downgrade to version 10.13. You may need to point to it using --with-macos-sdk=<path> in your mozconfig. Various SDK versions are available from https://github.com/phracker/MacOSX-SDKs

OK, I grabbed SDK 10.13 from https://github.com/phracker/MacOSX-SDKs as recommended by the error message, and added the appropriate --with-macos-sdk to a mozconfig.

So now configure is happy; but my build fails:

 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/xpcom/io/nsLocalFileUnix.cpp:45:
 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/xpcom/io/nsLocalFile.h:44:
 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/xpcom/io/nsLocalFileUnix.h:24:
 7:27.22 In file included from /Users/jkew/mozdev/mozilla-unified/obj-x86_64-apple-darwin17.7.0/dist/include/nsILocalFileMac.h:17:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:23:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:87:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/Components.h:26:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/Files.h:56:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/DiskArbitration.framework/Headers/DADisk.h:28:
 7:27.22 In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/IOKit.framework/Headers/IOKitLib.h:49:
 7:27.22 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/IOKit.framework/Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 7:27.22 #include <IOKit/IOReturn.h>
 7:27.22          ^~~~~~~~~~~~~~~~~~
 7:28.17 BUILDSTATUS BUILD_VERBOSE xpcom/base
 7:28.19 1 error generated.
 7:28.21 make[4]: *** [nsLocalFileUnix.o] Error 1

Any idea what's wrong?

Flags: needinfo?(mshal)

I'm trying a build now with sdk_max_version bumped to 10.14, I expect that to give me the same results as I used to get before this landed. Will confirm when it's done.

Yes, that works, but it's a bit annoying to have to use a local build system patch on what ought to be a pretty straightforward development configuration.

My build with that patch failed:

31:51.17 In file included from /Users/kats/zspace/gecko-mac/obj-host-opt/widget/cocoa/Unified_mm_widget_cocoa0.mm:11:
31:51.17 /Users/kats/zspace/gecko-mac/widget/cocoa/GfxInfo.mm:78:34: warning: 'CGDisplayIOServicePort' is deprecated: first deprecated in macOS 10.9 - No longer supported [-Wdeprecated-declarations]
31:51.17   io_registry_entry_t dsp_port = CGDisplayIOServicePort(kCGDirectMainDisplay);
31:51.17                                  ^
31:51.18 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreGraphics.framework/Headers/CGDisplayConfiguration.h:372:24: note: 'CGDisplayIOServicePort' has been explicitly marked deprecated here
31:51.18 CG_EXTERN io_service_t CGDisplayIOServicePort(CGDirectDisplayID display)
31:51.18                        ^
31:51.18 In file included from /Users/kats/zspace/gecko-mac/obj-host-opt/widget/cocoa/Unified_mm_widget_cocoa0.mm:110:
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsAppShell.mm:316:15: warning: 'loadNibFile:externalNameTable:withZone:' is deprecated: first deprecated in macOS 10.8 [-Wdeprecated-declarations]
31:51.19     [NSBundle loadNibFile:[NSString stringWithUTF8String:(const char*)nibPath.get()]
31:51.19               ^
31:51.19 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/AppKit.framework/Headers/NSNibLoading.h:30:1: note: 'loadNibFile:externalNameTable:withZone:' has been explicitly marked deprecated here
31:51.19 + (BOOL)loadNibFile:(null_unspecified NSString *)fileName externalNameTable:(null_unspecified NSDictionary *)context withZone:(null_unspecified NSZone *)zone NS_DEPRECATED_MAC(10_0, 10_8);
31:51.19 ^
31:51.19 In file included from /Users/kats/zspace/gecko-mac/obj-host-opt/widget/cocoa/Unified_mm_widget_cocoa0.mm:137:
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1114:24: error: 'authorizationStatusForMediaType:' is only available on macOS 10.14 or newer [-Werror,-Wunguarded-availability-new]
31:51.19       [AVCaptureDevice authorizationStatusForMediaType:aMediaType]);
31:51.19                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/AVFoundation.framework/Headers/AVCaptureDevice.h:1479:1: note: 'authorizationStatusForMediaType:' has been explicitly marked partial here
31:51.19 + (AVAuthorizationStatus)authorizationStatusForMediaType:(AVMediaType)mediaType API_AVAILABLE(macos(10.14), ios(7.0));
31:51.19 ^
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1114:24: note: enclose 'authorizationStatusForMediaType:' in an @available check to silence this warning
31:51.19       [AVCaptureDevice authorizationStatusForMediaType:aMediaType]);
31:51.19                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1209:20: error: 'requestAccessForMediaType:completionHandler:' is only available on macOS 10.14 or newer [-Werror,-Wunguarded-availability-new]
31:51.19   [AVCaptureDevice requestAccessForMediaType:aType completionHandler:aHandler];
31:51.19                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/AVFoundation.framework/Headers/AVCaptureDevice.h:1502:1: note: 'requestAccessForMediaType:completionHandler:' has been explicitly marked partial here
31:51.19 + (void)requestAccessForMediaType:(AVMediaType)mediaType completionHandler:(void (^)(BOOL granted))handler API_AVAILABLE(macos(10.14), ios(7.0));
31:51.19 ^
31:51.19 /Users/kats/zspace/gecko-mac/widget/cocoa/nsCocoaUtils.mm:1209:20: note: enclose 'requestAccessForMediaType:completionHandler:' in an @available check to silence this warning
31:51.19   [AVCaptureDevice requestAccessForMediaType:aType completionHandler:aHandler];
31:51.19                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
31:51.19 2 warnings and 2 errors generated.
31:51.19 make[4]: *** [Unified_mm_widget_cocoa0.o] Error 1
31:51.19 make[4]: *** Waiting for unfinished jobs....
31:51.19 editor/txmgr/tests
31:51.58 editor/composer
31:53.60 layout/style
31:57.70 layout/style/test/gtest
31:59.11 make[3]: *** [widget/cocoa/target] Error 2
31:59.11 make[3]: *** Waiting for unfinished jobs....
32:11.56 make[2]: *** [compile] Error 2
32:11.56 make[1]: *** [default] Error 2
32:11.56 make: *** [build] Error 2

I filed bug 1544418 to track this; let's continue discussion over there.

Reporter

Updated

a month ago
Flags: needinfo?(mshal)
Regressions: 1544732
Reporter

Comment 25

a month ago

This was backed out by bug 1544418 - :spohl, I could use some guidance on what the current set of acceptable SDKs for local developers is, and maybe some help on why adding the check results in failures on otherwise good versions. (eg: https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c17 and https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c18)

Status: RESOLVED → REOPENED
Flags: needinfo?(spohl.mozilla.bugs)
Resolution: FIXED → ---

(In reply to Michael Shal [:mshal] from comment #25)

This was backed out by bug 1544418 - :spohl, I could use some guidance on what the current set of acceptable SDKs for local developers is, and maybe some help on why adding the check results in failures on otherwise good versions. (eg: https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c17 and https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c18)

The only SDK that we officially support at the moment is the 10.11 SDK. I will not have the bandwidth to test various SDK/macOS combinations. If a developer encounters issues with an SDK other than 10.11 (for example visual glitches), the first step should be to rebuild with the 10.11 SDK.

If building with the 10.14 SDK is now working and no longer results in completely black windows we can leave this SDK check removed. We should be ready to reintroduce a similar check if a future SDK were to break things again.

Flags: needinfo?(spohl.mozilla.bugs)
Reporter

Updated

a month ago
Assignee: mshal → nobody

To anyone else who's a bit concerned about the unverified nature of the phracker SDK repository:

You can download an official copy of the 10.11 SDK by downloading Xcode 7.1.1 from apple's developer portal [1] [2]. Mount the DMG, right-click on the installer, go to "show package contents". Then go Contents->Developer->Platforms->MacOSX.platform->Developer->SDKs to find MacOSX10.11.sdk.

I have verified that this very-nearly matches this 10.11 SDK on phracker: [3] [4] (verify the checksum!). The differences between the two are available for perusal at [5], and do not appear to be malicious.

[1] https://download.developer.apple.com/Developer_Tools/Xcode_7.1.1/Xcode_7.1.1.dmg
[2] SHA1: 33163b3cbff846224a2661c08ef8e737f0da0762
[3] https://github.com/phracker/MacOSX-SDKs/releases/download/MacOSX10.11.sdk/MacOSX10.11.sdk.tar.xz
[4] SHA1: f3430e3d923644e66c0c13f7a48754e7b6aa2e3f
[5] https://gist.github.com/bholley/4ba2e619912df44d97be7d8634921e20

You need to log in before you can comment on or make changes to this bug.