1696513 - Increase minimum required macOS SDK version to the macOS 11 SDK

Status: RESOLVED FIXED

(Firefox Build System :: General, task, P3)

Description

[Markus Stange [:mstange]]

Once we build all macOS variants of Firefox with the macOS 11 SDK in CI (bug 1696504), and once we've given developers enough of a heads-up to update their macOS version or obtain the macOS 11 SDK, we should increase our minimum required SDK version to the macOS 11 SDK.

We will no longer have coverage of older SDKs in CI, so it would be easy to accidentally introduce breakage for developers who build with older SDKs.

Increasing the minimum required SDK will unlock the following improvements for Firefox macOS development:

• We will not need to worry about breaking SDKs that we don't have CI coverage for.
• Taking advantage of modern macOS APIs will be much easier. See Using new APIs with old SDKs for a list of workarounds that we currently have to employ. These workarounds can be a serious impediment to getting things done - see bug 1671626 comment 8 for an example where we had trouble using the MediaPlayer framework before we were building with the 10.12 SDK.
• We can remove the existing workarounds, i.e. any code that's only needed for old SDKs..
• The compiler will catch any missing runtime version checks of the form if (@available(macOS 10.14, *)) { when accessing modern APIs, because the header definitions from the modern SDKs are annotated with the correct API_AVAILABLE attributes. (This would have caught bug 1602813, for example.)

Comment 1

[Harry Twyford]

We should remember to update the SDK section of the macOS docs when we make this change.

Comment 2

[Markus Stange [:mstange]]

Bug 1759504 just landed and effectively increased our minimum required SDK version to 10.13. But we don't check for the 10.13 SDK at configure time.

Comment 3

[Stephen A Pohl [:spohl]]

Attachment: Bug 1696513: Require macOS 11 SDK at configure time and update docs. r=mstange

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f636b7e35d0c
Require macOS 11 SDK at configure time and update docs. r=mstange

Comment 5

[Cristina Cozmuta (:CrissCozmuta)]

Backed out for causing build bustages. CLOSED TREE

Backout link
Push with failures
Link to failure log
Failure line : /builds/worker/checkouts/gecko/gfx/thebes/gfxMacUtils.cpp:16:14: error: 'kCVImageBufferTransferFunction_sRGB' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]

Link to failure log 2
Failure line : python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py::OSXToolchainTest::test_clang TEST-UNEXPECTED-FAIL

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e186162443a6
Require macOS 11 SDK at configure time and update docs. r=mstange

Comment 7

[Stephen A Pohl [:spohl]]

It looks like this is about to get backed out again for this error:

ERROR: SDK version "10.15.6" is too old. Please upgrade to at least 11.0. Try updating your system Xcode. If that's not sufficient, see the manual SDK installation docs: https://firefox-source-docs.mozilla.org/setup/macos_build.html#macos-sdk-is-unsupported
https://firefoxci.taskcluster-artifacts.net/DqFxpIDCTqCZhTZ1XfVuvQ/0/public/logs/live_backing.log

Glandium, I thought we had updated our build machines to use the macOS 11 SDK. Am I missing something..?

Comment 8

[Cristina Cozmuta (:CrissCozmuta)]

Hi Stephen,
These builds don't normally run on autoland and were triggered by you.
They only run on central from what I can see and they have no tests attached to them.
https://tinyurl.com/2aef7sxk

Backed out for causing bustages.
Backout link
Push with failures
Link to failure log

Comment 9

[Stephen A Pohl [:spohl]]

(In reply to Cristina Cozmuta (:CrissCozmuta) from comment #8)

Hi Stephen,
These builds don't normally run on autoland and were triggered by you.

Right, I triggered these because I was getting worried about failures that I was seeing on try and I didn't want this to make it to m-c if there was going to be bustage. Thanks for backing out the patch and I'll wait to hear back from glandium.

Comment 10

[mac198442]

(In reply to Stephen A Pohl [:spohl] from comment #7)

It looks like this is about to get backed out again for this error:

ERROR: SDK version "10.15.6" is too old. Please upgrade to at least 11.0. Try updating your system Xcode. If that's not sufficient, see the manual SDK installation docs: https://firefox-source-docs.mozilla.org/setup/macos_build.html#macos-sdk-is-unsupported
https://firefoxci.taskcluster-artifacts.net/DqFxpIDCTqCZhTZ1XfVuvQ/0/public/logs/live_backing.log

Glandium, I thought we had updated our build machines to use the macOS 11 SDK. Am I missing something..?

As far as I know, the automation builds use SDK 10.15.

Comment 11

[mac198442]

(In reply to mac198442 from comment #10)

(In reply to Stephen A Pohl [:spohl] from comment #7)

It looks like this is about to get backed out again for this error:

ERROR: SDK version "10.15.6" is too old. Please upgrade to at least 11.0. Try updating your system Xcode. If that's not sufficient, see the manual SDK installation docs: https://firefox-source-docs.mozilla.org/setup/macos_build.html#macos-sdk-is-unsupported
https://firefoxci.taskcluster-artifacts.net/DqFxpIDCTqCZhTZ1XfVuvQ/0/public/logs/live_backing.log

Glandium, I thought we had updated our build machines to use the macOS 11 SDK. Am I missing something..?

As far as I know, the automation builds use SDK 10.15.

I guess I should have checked they are using 11.0

Comment 12

[Mike Hommey [:glandium]]

They are using whatever xcrun --show-sdk-path returns. This is kind of highlighting that configure should probably try harder to find an adequate SDK because those machines do have more recent SDKs installed.

Comment 13

[Stephen A Pohl [:spohl]]

We should probably ensure that all our builds are built using the same version of the SDK (we'd expect builds to be built with the macOS 11 SDK at the moment). Could we explicitly set the Xcode that is being used using xcode-select? Can this happen during configure before xcrun --show-sdk-path? Is it safe to assume that the Xcode containing the macOS 11 SDK is installed at a predictable path? Bug 1686129 would make it seem so, but not sure what specific Xcode is actually installed on these machines.

Comment 14

[Mike Hommey [:glandium]]

We should probably ensure that all our builds are built using the same version of the SDK (we'd expect builds to be built with the macOS 11 SDK at the moment).

I don't think it matters /that/ much for the "plain" builds, as long as they use a SDK that is deemed usable. Those builds are not shipped, they're there to ensure things would likely work on developer's machines.

Could we explicitly set the Xcode that is being used using xcode-select?

I don't think we should...

Can this happen during configure before xcrun --show-sdk-path?

... especially not during configure. We don't want to enforce arbitrary changes on people's configuration during configure.

The funny thing is, on the workers (and potentially on some people's machines), xcrun --show-sdk-path doesn't even pick the newest SDK it can. Case in point: it currently picks /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk, but:

• /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk exists
• /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk points to it!

Thus comment 12.

Comment 15

[Stephen A Pohl [:spohl]]

Okay, I understand "what" we need to do (comment 12), but it isn't clear to me "how" to do this. Could you clarify? Ideally, I'd like to land the patch in this bug soon. (I tried pinging you on Matrix. We could discuss there if you prefer.)

(In reply to Mike Hommey [:glandium] from comment #14)

We should probably ensure that all our builds are built using the same version of the SDK (we'd expect builds to be built with the macOS 11 SDK at the moment).

I don't think it matters /that/ much for the "plain" builds, as long as they use a SDK that is deemed usable. Those builds are not shipped, they're there to ensure things would likely work on developer's machines.

True, it may not matter too much at this time because the resulting build "should" be identical, but since we want to require developer builds to be built with a macOS 11 SDK or higher, we should probably enforce the same here.

Could we explicitly set the Xcode that is being used using xcode-select?

I don't think we should...

Does your statement apply to our build machines too? It would seem that we would want to be explicit about what SDK is being used.

Can this happen during configure before xcrun --show-sdk-path?

... especially not during configure. We don't want to enforce arbitrary changes on people's configuration during configure.

Fair enough. Is there a different point where we could explicitly set the SDK for our build machines?

The funny thing is, on the workers (and potentially on some people's machines), xcrun --show-sdk-path doesn't even pick the newest SDK it can. Case in point: it currently picks /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk, but:

• /Library/Developer/CommandLineTools/SDKs/MacOSX11.0.sdk exists
• /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk points to it!

Thus comment 12.

My understanding is that the SDK that is being returned by xcrun --show-sdk-path was previously set by xcode-select, but that this can be executed by an Xcode install for example. So if a version of Xcode with the macOS 10.15 SDK was INSTALLED, but separate SDKs are then COPIED to the system, xcrun --show-sdk-path will still default to the macOS 10.15 SDK from the Xcode install.

Comment 16

[Stephen A Pohl [:spohl]]

We discussed this in the macOS Spotlight meeting and think that it is safe for the patch to land again. On the build systems that are not usually run by default, we would want to run xcode-select to point to the macOS 11 SDK, and this should address the issue going forward.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e69b42e2fe0c
Require macOS 11 SDK at configure time and update docs. r=mstange

Comment 18

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/e69b42e2fe0c

Comment 19

[Alex Finder]

(In reply to Cristina Cozmuta (:CrissCozmuta) from comment #8)

Hi Stephen,
These builds don't normally run on autoland and were triggered by you.
They only run on central from what I can see and they have no tests attached to them.
https://tinyurl.com/2aef7sxk

Backed out for causing bustages.
Backout link
Push with failures
Link to failure log

== Change summary for alert #34470 (as of Tue, 14 Jun 2022 13:54:36 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
10%	expedia ContentfulSpeedIndex	macosx1015-64-shippable-qr	cold fission webrender	449.71 -> 494.42
7%	expedia ContentfulSpeedIndex	macosx1015-64-shippable-qr	cold fission webrender	460.12 -> 493.58
4%	expedia loadtime	macosx1015-64-shippable-qr	cold fission webrender	1,255.12 -> 1,309.83
4%	expedia loadtime	macosx1015-64-shippable-qr	cold fission webrender	1,252.79 -> 1,305.92
4%	expedia PerceptualSpeedIndex	macosx1015-64-shippable-qr	cold fission webrender	982.89 -> 1,021.08
3%	expedia PerceptualSpeedIndex	macosx1015-64-shippable-qr	cold fission webrender	986.88 -> 1,021.17

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=34470