1799312 - macos_build - improve MacOS SDK setup / documentation. Why require 30 GB if 600 MB works.

Status: RESOLVED WORKSFORME

(Firefox Build System :: Build Documentation, task)

Description

[Rob Wu [:robwu]]

To build Firefox on macOS, a SDK from XCode is needed. It is not automatically installed with ./mach bootstrap, and the instructions from the documentation are also quite demanding. I followed the documented instructions and needed 30 GB disk space and 11 GB network usage, while exactly the same result can be achieved with merely 600 MB disk space (and 50 MB network usage), by only downloading an archive of the SDK (I used MacOSX11.3.sdk.tar.xz). More details on the experience and verification are below.

Ideally the SDK would be installed by ./mach bootstrap, but if that's infeasible, then it would be nice to improve the documentation.

The documented requirement is currently MacOS SDK 11.3 from Xcode 12.5.1.
I see that there is a bug (bug 1773743) for bumping to MacOS SDK 13 from Xcode 14, where a similar issue is going to happen, but at a larger scale.

Detailed version

./mach build reports

ERROR: SDK version "11.1" is too old. Please upgrade to at least 11.3. Try updating your system Xcode. If that's not sufficient, see the manual SDK installation docs: https://firefox-source-docs.mozilla.org/setup/macos_build.html#macos-sdk-is-unsupported

That macos_build documentation shows:

Go to the More Downloads for Apple Developers page and download Xcode 12.5.1.

This requires a log in with an Apple ID, which unnecessarily raises the bar for developing Firefox on macOS. I didn't bother with that, and found a copy of Xcode 12.5.1 at Archive.org, to Xcode_12.5.1.xip, of 11248 MB. After creating a directory and extracting it with xip --expand ../Xcode_12.5.1.xip, the current directory was replaced with a directory containing Xcode.app, of 15986 MB (measured with du -m; actually larger, but smaller due to transparent APFS compression). So the minimum disk requirement to set this up is about 30 GB (27 GB).
(Without enough disk space, I get xip: error: The archive “Xcode_12.5.1.xip” can’t be expanded because the selected volume doesn’t have enough free space.)

The current 30 GB disk and 11 GB network requirements are excessive, especially because the MacOSX11.3.sdk is 575 MB (244 MB with filesystem compression, which appears to be used when I extracted with xip --expand). The current instructions from the Fx source docs suggest to copy with cp -aH, which results in twice the disk usage. To have a smaller disk usage, use ditto --hfsCompression instead of cp -aH, as in: ditto --hfsCompression ~/Downloads/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk ~/.mozbuild/macos-sdk/.

Significant savings can be achieved by avoiding Xcode and only downloading a copy of the SDK, e.g. from https://github.com/phracker/MacOSX-SDKs/releases/download/11.3/MacOSX11.3.sdk.tar.xz (part of https://github.com/phracker/MacOSX-SDKs) and extract with tar xf MacOSX11.3.sdk.tar.xz. I confirmed with diff -r dir1 dir2 that the contents of the SDK from Xcode and MacOSX11.3.sdk.tar.xz are identical. These are the file hashes:

$ sha -a 256 MacOSX11.3.sdk.tar.xz
cd4f08a75577145b8f05245a2975f7c81401d75e9535dcffbb879ee1deefcbf4 MacOSX11.3.sdk.tar.xz
$ md5 MacOSX11.3.sdk.tar.xz
MD5 (MacOSX11.3.sdk.tar.xz) = 22c0ed904531f6d7dee4f87c15f7cf47

Comment 1

[Mike Hommey [:glandium]]

The legal status of those SDKs is dubious at best. We can't recommend people use them.

Comment 2

[Rob Wu [:robwu]]

Allegedly an alternative is to download the "Command Line Tool" packages, according to https://github.com/phracker/MacOSX-SDKs/issues/39#issuecomment-955144711

I have not personally looked for that (again, because https://developer.apple.com/download/ requires a login), so I cannot verify whether such a package exists.

Could the macos_build documentation at least be updated to note that Xcode is not necessarily required, perhaps with a reference to this bug for now until a better solution comes along?

Comment 3

[Rob Wu [:robwu]]

Fixed by bug 1522986, specifically https://hg.mozilla.org/integration/autoland/rev/697accbb8a41. That removes the documentation with manual steps, in favor of automatically installing the necessary dependency via ./mach bootstrap. Only the necessary SDK is downloaded, not the full Xcode app that contains it.