1560044 - ./mach build fails with config/pathsub.c:10:10: fatal error: 'assert.h' file not found

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect, P1)

Description

[Michael Froman [:mjf]]

I'm on macOS 10.14.5, with latest moz-central pulled (as of 2019/06/18).
Working through the instructions on https://mozilla.github.io/geckoview/contributor/geckoview-quick-start gives me the error:

0:01.09 /Users/mfroman/mozilla/moz-central/config/pathsub.c:10:10: fatal error: 'assert.h' file not found
0:01.09 #include <assert.h>
0:01.09 ^~~~~~~~~~
0:01.09 /Users/mfroman/mozilla/moz-central/config/nsinstall.c:10:10: fatal error: 'stdio.h' file not found
0:01.09 #include <stdio.h> /* OSF/1 requires this before grp.h, so put it first */
0:01.09 ^~~~~~~~~
0:01.09 1 error generated.
0:01.09 1 error generated.

I've tried uncommenting each of the targets given in the example mozconfig during bootstrap, including no target uncommented. All give the same results.

Comment 1

[Nick Alexander :nalexander [he/him]]

etoop: this is what you witnessed, right?

mjf: something happens with macOS removing the Xcode headers. You can manually re-install the Xcode headers; I think etoop has the command and hopefully will post it here.

Comment 2

[Nick Alexander :nalexander [he/him]]

mjf: something happens with macOS removing the Xcode headers. You can manually re-install the Xcode headers; I think etoop has the command and hopefully will post it here.

mjf: try https://bugzilla.mozilla.org/show_bug.cgi?id=1487552#c55 and let us know if that helps?

Comment 3

[Emily Toop (:fluffyemily)]

The command is:
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

If you get the error that the file does not exist, you have to download the command line tools from the Apple website and reinstall them and then run the command.

Comment 4

[Michael Froman [:mjf]]

I re-downloaded the command-line tools from Apple. For folks playing along at home, this link is very useful: https://developer.apple.com/download/more/ (then search for "command line tools"). Reinstalling Command Line Tools (macOS Mojave version 10.14).pkg and then the command from Comment 3 does seem to fix the issue (build is running now).

Bug 1526857 landed a couple weeks ago and seems to have fixed this issue for normal macOS builds. Could a similar fix for the android build reduce the pain for future Android devs?

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1560044 - Allow --with-macos-sdk as an option when cross-compiling on macOS.

Building host tools on macOS require a macOS SDK, but it's currently
not configurable when cross-compiling for e.g. Android.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1560044 - Always use a SDK path on macOS.

The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
which is not really sustainable.

This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.

Comment 7

[Mike Hommey [:glandium]]

Attachment: Bug 1560044 - Make Xcode checks macOS host only.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7ae0d606f1c6
Allow --with-macos-sdk as an option when cross-compiling on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/a5bbaa8c1001
Always use a SDK path on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/1271940e91e8
Make Xcode checks macOS host only. r=nalexander

Comment 9

[Andreea Pavel [:apavel]]

Backed out for build bustages

Push with failures https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=1271940e91e86515e9e4f0429206b98475931c5c&selectedJob=254500163

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=254500163&repo=autoland&lineNumber=42275

Backout: https://hg.mozilla.org/integration/autoland/rev/53b7325a470a96c4ec659d57d6465308dba02020

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9114bef9e25a
Allow --with-macos-sdk as an option when cross-compiling on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/3fd5b373540b
Always use a SDK path on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/715b1861e5ac
Make Xcode checks macOS host only. r=nalexander

Comment 11

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/9114bef9e25a
https://hg.mozilla.org/mozilla-central/rev/3fd5b373540b
https://hg.mozilla.org/mozilla-central/rev/715b1861e5ac

Comment 12

[Jonathan Kew [:jfkthame]]

This has broken my local build on macOS 10.13:

 0:50.14 DEBUG: In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOKitLib.h:49:
 0:50.14 DEBUG: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 0:50.14 DEBUG: #include <IOKit/IOReturn.h>
 0:50.14 DEBUG:          ^~~~~~~~~~~~~~~~~~
 0:50.14 DEBUG: 1 error generated.

We saw something like this previously, IIRC, but offhand I can't remember exactly what the fix/workaround was... :-(

Comment 13

[Jonathan Kew [:jfkthame]]

Oh yes, bug 1544763 (and see also bug 1544418). So maybe I can switch to the 10.14 sdk...

Comment 14

[Chris Peterson [:cpeterson]]

Mike, Fennec is moving to maintenance mode in the ESR 68 branch. There will be no Fennec 69 release. Should we uplift your macOS configure fix to the ESR 68 branch for engineers that will need to build Fennec ESR 68 for the next 12 months?

Comment 15

[Mike Hommey [:glandium]]

(In reply to Jonathan Kew (:jfkthame) from comment #12)

This has broken my local build on macOS 10.13:

 0:50.14 DEBUG: In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOKitLib.h:49:
 0:50.14 DEBUG: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 0:50.14 DEBUG: #include <IOKit/IOReturn.h>
 0:50.14 DEBUG:          ^~~~~~~~~~~~~~~~~~
 0:50.14 DEBUG: 1 error generated.

We saw something like this previously, IIRC, but offhand I can't remember exactly what the fix/workaround was... :-(

Please file a followup bug, with more details.

(In reply to Chris Peterson [:cpeterson] from comment #14)

Mike, Fennec is moving to maintenance mode in the ESR 68 branch. There will be no Fennec 69 release. Should we uplift your macOS configure fix to the ESR 68 branch for engineers that will need to build Fennec ESR 68 for the next 12 months?

Maybe, but not before the above and bug 1564216 are sorted out.

Comment 16

[Mike Hommey [:glandium]]

Please file a followup bug, with more details.

That said, I suspect your SDK doesn't come from Xcode. At least, I tried with Xcode 9.4.1, which is the last that seems to come with the 10.13 SDK, and have not been able to reproduce with it.

Comment 17

[Jonathan Kew [:jfkthame]]

(In reply to Mike Hommey [:glandium] from comment #16)

Please file a followup bug, with more details.

That said, I suspect your SDK doesn't come from Xcode. At least, I tried with Xcode 9.4.1, which is the last that seems to come with the 10.13 SDK, and have not been able to reproduce with it.

I believe (if memory serves) you're correct; my Xcode came with the 10.14 SDK. When first installed, this worked fine for building Firefox. I didn't need to select any explicit SDK in my mozconfig, it just worked.

Then the build system was modified in bug 1522931 to check the version of the default SDK; this made the build error out on the 10.14 SDK.

So I downgraded to the 10.13 SDK using the package from https://github.com/phracker/MacOSX-SDKs/releases/tag/10.13, because that's what mach advised in its error message.

Unfortunately, it turns out (according to :kats in bug 1544763 comment 6) that the tarball there has broken links, which leads to the IOKit/IOReturn.h failure.

I think that since bug 1522931 was backed out/reopened, I've been building again with my default Xcode-provided 10.14 SDK; but then the patches here have switched me back to the (broken) 10.13 SDK from phracker, thereby resurrecting bug 1544763.

There's enough information around these various bugs that I can find ways to resolve the issue here -- I believe reinstalling the 10.13 SDK (not from the phracker tarball) should avoid the problem, or else I can add --with-macos-sdk= ...SDKs/MacOSX10.14.sdk/ in my mozconfig, which is what I've done -- but it's a bit disconcerting when rebasing a stack of patches to current m-c suddenly breaks the build in non-obvious ways; trying to figure out what broke led me to the patches here.

Comment 18

[Mike Hommey [:glandium]]

Or you can reinstall the 10.14 SDK.

Comment 19

[Mike Hommey [:glandium]]

Comment on attachment 9075286 [details]
Bug 1560044 - Allow --with-macos-sdk as an option when cross-compiling on macOS.

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fennec is moving to maintenance mode in the ESR 68 branch. There will be no Fennec 69 release. We should uplift these patches to the ESR 68 branch for engineers that will need to build Fennec ESR 68 for the next 12 months.
• User impact if declined: Fennec developers using a mac having to go through hoops to build Fennec on their mac.
• Fix Landed on Version: 69
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Configure-time changes only, that only affects defaults. The changes only affect builds on or for mac, so for shipping builds, that would affect cross-mac builds, but those are not using defaults anyways, so they're actually not affected. There are some consequence for developer builds, as the followups to this bug have shown, but all have been identified to be caused by local footguns (--enable-warnings-as-errors and bad SDKs).
  Note there is a conflict in test_toolchain_configure.py when applying the patch on esr68, but it's only a context conflict that is trivially fixable.
• String or UUID changes made by this patch: N/A

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9075286 [details]
Bug 1560044 - Allow --with-macos-sdk as an option when cross-compiling on macOS.

Fixes Fennec build bustage when building on macOS. Approved for ESR68.

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-esr68/rev/ee9e4771f9b8
https://hg.mozilla.org/releases/mozilla-esr68/rev/47d890eaa341
https://hg.mozilla.org/releases/mozilla-esr68/rev/4fa7260201bb

Comment 22

[Nick Alexander :nalexander [he/him]]

Clearing NI as glandium has done great work addressing this. Thanks, glandium!

Comment 23

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for linux build test failure at test_toolchain_configure.py::OSXToolchainTest::test_unsupported_clang:

https://hg.mozilla.org/releases/mozilla-esr68/rev/726a3249f18ba4f847a291d427acd1cad41c5763

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr68&resultStatus=testfailed%2Cbusted%2Cexception&revision=4fa7260201bb5cb3708f914d1766c1d35917a5ee
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=257184343&repo=mozilla-esr68

[task 2019-07-18T16:22:39.571Z] 16:22:39 WARNING - ../python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py::OSXToolchainTest::test_unsupported_clang TEST-UNEXPECTED-FAIL

[...]

[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - =================================== FAILURES ===================================
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - ___________________ OSXToolchainTest.test_unsupported_clang ____________________
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - self = <test_toolchain_configure.OSXToolchainTest testMethod=test_unsupported_clang>
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - def test_unsupported_clang(self):
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - # clang 3.3 C compiler is perfectly fine, but we need more for C++.
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - self.do_toolchain_test(self.PATHS, {
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - 'c_compiler': self.CLANG_3_3_RESULT,
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - 'cxx_compiler': self.CLANGXX_3_3_RESULT,
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - }, environ={
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - 'CC': 'clang-3.3',
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - > 'CXX': 'clang++-3.3',
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - })
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - ../python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py:867:
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - ../python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py:384: in do_toolchain_test
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - self.assertEquals((var, compiler), (var, result))
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E AssertionError: Tuples differ: (u'c_compiler', <ReadOnlyNames... != (u'c_compiler', <CompilerResul...
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E First differing element 1:
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E <ReadOnlyNamespace {'language': u'C', 'wrapper': [], 'version': LooseVersion ('3.3.0'), 'flags': [u'-isysroot', u'/builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/configure/fake_macos_sdk'], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E <CompilerResult {'language': u'C', 'wrapper': [], 'version': u'3.3.0', 'flags': [], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E (u'c_compiler',
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - E - <ReadOnlyNamespace {'language': u'C', 'wrapper': [], 'version': LooseVersion ('3.3.0'), 'flags': [u'-isysroot', u'/builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/configure/fake_macos_sdk'], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>)
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - E + <CompilerResult {'language': u'C', 'wrapper': [], 'version': u'3.3.0', 'flags': [], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>)
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - ==================== 1 failed, 51 passed in 105.82 seconds =====================
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - Setting retcode to 1 from /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - Return code from mach python-test: 1
[task 2019-07-18T16:22:39.730Z] 16:22:39 INFO - /builds/worker/workspace/build/src/testing/testsuite-targets.mk:269: recipe for target 'check' failed
[task 2019-07-18T16:22:39.730Z] 16:22:39 INFO - make: *** [check] Error 1

Comment 24

[Mike Hommey [:glandium]]

Turns out the unit tests need bug 1557547 to work. Alternatively, we can change the unit test patch to make it work, but I think uplifting bug 1557547 is low-risk enough and useful enough.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/b7e8b83d86da
https://hg.mozilla.org/releases/mozilla-esr68/rev/7be1a6c9e9bc
https://hg.mozilla.org/releases/mozilla-esr68/rev/9c631c20c989