./mach build fails with config/pathsub.c:10:10: fatal error: 'assert.h' file not found

RESOLVED FIXED in Firefox -esr68

Status

defect
P1
normal
RESOLVED FIXED
2 months ago
29 days ago

People

(Reporter: mjf, Assigned: glandium)

Tracking

(Blocks 1 bug)

unspecified
mozilla69
Unspecified
macOS
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 wontfix, firefox-esr68 fixed, firefox68 wontfix, firefox69 fixed)

Details

Attachments

(3 attachments)

I'm on macOS 10.14.5, with latest moz-central pulled (as of 2019/06/18).
Working through the instructions on https://mozilla.github.io/geckoview/contributor/geckoview-quick-start gives me the error:

0:01.09 /Users/mfroman/mozilla/moz-central/config/pathsub.c:10:10: fatal error: 'assert.h' file not found
0:01.09 #include <assert.h>
0:01.09 ^~~~~~~~~~
0:01.09 /Users/mfroman/mozilla/moz-central/config/nsinstall.c:10:10: fatal error: 'stdio.h' file not found
0:01.09 #include <stdio.h> /* OSF/1 requires this before grp.h, so put it first */
0:01.09 ^~~~~~~~~
0:01.09 1 error generated.
0:01.09 1 error generated.

I've tried uncommenting each of the targets given in the example mozconfig during bootstrap, including no target uncommented. All give the same results.

etoop: this is what you witnessed, right?

mjf: something happens with macOS removing the Xcode headers. You can manually re-install the Xcode headers; I think etoop has the command and hopefully will post it here.

Flags: needinfo?(etoop)

mjf: something happens with macOS removing the Xcode headers. You can manually re-install the Xcode headers; I think etoop has the command and hopefully will post it here.

mjf: try https://bugzilla.mozilla.org/show_bug.cgi?id=1487552#c55 and let us know if that helps?

Flags: needinfo?(mfroman)

The command is:
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

If you get the error that the file does not exist, you have to download the command line tools from the Apple website and reinstall them and then run the command.

Flags: needinfo?(etoop)

I re-downloaded the command-line tools from Apple. For folks playing along at home, this link is very useful: https://developer.apple.com/download/more/ (then search for "command line tools"). Reinstalling Command Line Tools (macOS Mojave version 10.14).pkg and then the command from Comment 3 does seem to fix the issue (build is running now).

Bug 1526857 landed a couple weeks ago and seems to have fixed this issue for normal macOS builds. Could a similar fix for the android build reduce the pain for future Android devs?

Flags: needinfo?(mfroman) → needinfo?(nalexander)

Building host tools on macOS require a macOS SDK, but it's currently
not configurable when cross-compiling for e.g. Android.

The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
which is not really sustainable.

This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.

Assignee: nobody → mh+mozilla
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/7ae0d606f1c6
Allow --with-macos-sdk as an option when cross-compiling on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/a5bbaa8c1001
Always use a SDK path on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/1271940e91e8
Make Xcode checks macOS host only. r=nalexander
Flags: needinfo?(mh+mozilla)
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/9114bef9e25a
Allow --with-macos-sdk as an option when cross-compiling on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/3fd5b373540b
Always use a SDK path on macOS. r=nalexander
https://hg.mozilla.org/integration/autoland/rev/715b1861e5ac
Make Xcode checks macOS host only. r=nalexander
Flags: needinfo?(mh+mozilla)

This has broken my local build on macOS 10.13:

 0:50.14 DEBUG: In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOKitLib.h:49:
 0:50.14 DEBUG: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 0:50.14 DEBUG: #include <IOKit/IOReturn.h>
 0:50.14 DEBUG:          ^~~~~~~~~~~~~~~~~~
 0:50.14 DEBUG: 1 error generated.

We saw something like this previously, IIRC, but offhand I can't remember exactly what the fix/workaround was... :-(

Flags: needinfo?(mh+mozilla)

Oh yes, bug 1544763 (and see also bug 1544418). So maybe I can switch to the 10.14 sdk...

Mike, Fennec is moving to maintenance mode in the ESR 68 branch. There will be no Fennec 69 release. Should we uplift your macOS configure fix to the ESR 68 branch for engineers that will need to build Fennec ESR 68 for the next 12 months?

OS: Unspecified → macOS
Priority: -- → P1
Regressions: 1564216

(In reply to Jonathan Kew (:jfkthame) from comment #12)

This has broken my local build on macOS 10.13:

 0:50.14 DEBUG: In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOKitLib.h:49:
 0:50.14 DEBUG: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/IOKit.framework/Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 0:50.14 DEBUG: #include <IOKit/IOReturn.h>
 0:50.14 DEBUG:          ^~~~~~~~~~~~~~~~~~
 0:50.14 DEBUG: 1 error generated.

We saw something like this previously, IIRC, but offhand I can't remember exactly what the fix/workaround was... :-(

Please file a followup bug, with more details.

(In reply to Chris Peterson [:cpeterson] from comment #14)

Mike, Fennec is moving to maintenance mode in the ESR 68 branch. There will be no Fennec 69 release. Should we uplift your macOS configure fix to the ESR 68 branch for engineers that will need to build Fennec ESR 68 for the next 12 months?

Maybe, but not before the above and bug 1564216 are sorted out.

Flags: needinfo?(mh+mozilla)

Please file a followup bug, with more details.

That said, I suspect your SDK doesn't come from Xcode. At least, I tried with Xcode 9.4.1, which is the last that seems to come with the 10.13 SDK, and have not been able to reproduce with it.

Flags: needinfo?(jfkthame)
Regressions: 1564298

(In reply to Mike Hommey [:glandium] from comment #16)

Please file a followup bug, with more details.

That said, I suspect your SDK doesn't come from Xcode. At least, I tried with Xcode 9.4.1, which is the last that seems to come with the 10.13 SDK, and have not been able to reproduce with it.

I believe (if memory serves) you're correct; my Xcode came with the 10.14 SDK. When first installed, this worked fine for building Firefox. I didn't need to select any explicit SDK in my mozconfig, it just worked.

Then the build system was modified in bug 1522931 to check the version of the default SDK; this made the build error out on the 10.14 SDK.

So I downgraded to the 10.13 SDK using the package from https://github.com/phracker/MacOSX-SDKs/releases/tag/10.13, because that's what mach advised in its error message.

Unfortunately, it turns out (according to :kats in bug 1544763 comment 6) that the tarball there has broken links, which leads to the IOKit/IOReturn.h failure.

I think that since bug 1522931 was backed out/reopened, I've been building again with my default Xcode-provided 10.14 SDK; but then the patches here have switched me back to the (broken) 10.13 SDK from phracker, thereby resurrecting bug 1544763.

There's enough information around these various bugs that I can find ways to resolve the issue here -- I believe reinstalling the 10.13 SDK (not from the phracker tarball) should avoid the problem, or else I can add --with-macos-sdk= ...SDKs/MacOSX10.14.sdk/ in my mozconfig, which is what I've done -- but it's a bit disconcerting when rebasing a stack of patches to current m-c suddenly breaks the build in non-obvious ways; trying to figure out what broke led me to the patches here.

Flags: needinfo?(jfkthame)
No longer regressions: 1564298

Or you can reinstall the 10.14 SDK.

Comment on attachment 9075286 [details]
Bug 1560044 - Allow --with-macos-sdk as an option when cross-compiling on macOS.

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fennec is moving to maintenance mode in the ESR 68 branch. There will be no Fennec 69 release. We should uplift these patches to the ESR 68 branch for engineers that will need to build Fennec ESR 68 for the next 12 months.
  • User impact if declined: Fennec developers using a mac having to go through hoops to build Fennec on their mac.
  • Fix Landed on Version: 69
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Configure-time changes only, that only affects defaults. The changes only affect builds on or for mac, so for shipping builds, that would affect cross-mac builds, but those are not using defaults anyways, so they're actually not affected. There are some consequence for developer builds, as the followups to this bug have shown, but all have been identified to be caused by local footguns (--enable-warnings-as-errors and bad SDKs).
    Note there is a conflict in test_toolchain_configure.py when applying the patch on esr68, but it's only a context conflict that is trivially fixable.
  • String or UUID changes made by this patch: N/A
Attachment #9075286 - Flags: approval-mozilla-esr68?
Attachment #9075287 - Flags: approval-mozilla-esr68?
Attachment #9075288 - Flags: approval-mozilla-esr68?

Comment on attachment 9075286 [details]
Bug 1560044 - Allow --with-macos-sdk as an option when cross-compiling on macOS.

Fixes Fennec build bustage when building on macOS. Approved for ESR68.

Attachment #9075286 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
Attachment #9075287 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
Attachment #9075288 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+

Clearing NI as glandium has done great work addressing this. Thanks, glandium!

Flags: needinfo?(nalexander)

Backed out for linux build test failure at test_toolchain_configure.py::OSXToolchainTest::test_unsupported_clang:

https://hg.mozilla.org/releases/mozilla-esr68/rev/726a3249f18ba4f847a291d427acd1cad41c5763

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr68&resultStatus=testfailed%2Cbusted%2Cexception&revision=4fa7260201bb5cb3708f914d1766c1d35917a5ee
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=257184343&repo=mozilla-esr68

[task 2019-07-18T16:22:39.571Z] 16:22:39 WARNING - ../python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py::OSXToolchainTest::test_unsupported_clang TEST-UNEXPECTED-FAIL

[...]

[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - =================================== FAILURES ===================================
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - ___________________ OSXToolchainTest.test_unsupported_clang ____________________
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - self = <test_toolchain_configure.OSXToolchainTest testMethod=test_unsupported_clang>
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - def test_unsupported_clang(self):
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - # clang 3.3 C compiler is perfectly fine, but we need more for C++.
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - self.do_toolchain_test(self.PATHS, {
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - 'c_compiler': self.CLANG_3_3_RESULT,
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - 'cxx_compiler': self.CLANGXX_3_3_RESULT,
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - }, environ={
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - 'CC': 'clang-3.3',
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - > 'CXX': 'clang++-3.3',
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - })
[task 2019-07-18T16:22:39.575Z] 16:22:39 INFO - ../python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py:867:
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - ../python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py:384: in do_toolchain_test
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - self.assertEquals((var, compiler), (var, result))
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E AssertionError: Tuples differ: (u'c_compiler', <ReadOnlyNames... != (u'c_compiler', <CompilerResul...
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E First differing element 1:
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E <ReadOnlyNamespace {'language': u'C', 'wrapper': [], 'version': LooseVersion ('3.3.0'), 'flags': [u'-isysroot', u'/builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/configure/fake_macos_sdk'], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E <CompilerResult {'language': u'C', 'wrapper': [], 'version': u'3.3.0', 'flags': [], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E
[task 2019-07-18T16:22:39.576Z] 16:22:39 INFO - E (u'c_compiler',
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - E - <ReadOnlyNamespace {'language': u'C', 'wrapper': [], 'version': LooseVersion ('3.3.0'), 'flags': [u'-isysroot', u'/builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/configure/fake_macos_sdk'], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>)
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - E + <CompilerResult {'language': u'C', 'wrapper': [], 'version': u'3.3.0', 'flags': [], 'type': u'clang', 'compiler': u'/usr/bin/clang-3.3'}>)
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - ==================== 1 failed, 51 passed in 105.82 seconds =====================
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - Setting retcode to 1 from /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/configure/test_toolchain_configure.py
[task 2019-07-18T16:22:39.577Z] 16:22:39 INFO - Return code from mach python-test: 1
[task 2019-07-18T16:22:39.730Z] 16:22:39 INFO - /builds/worker/workspace/build/src/testing/testsuite-targets.mk:269: recipe for target 'check' failed
[task 2019-07-18T16:22:39.730Z] 16:22:39 INFO - make: *** [check] Error 1

Flags: needinfo?(mh+mozilla)

Turns out the unit tests need bug 1557547 to work. Alternatively, we can change the unit test patch to make it work, but I think uplifting bug 1557547 is low-risk enough and useful enough.

Depends on: 1557547
Flags: needinfo?(mh+mozilla)
Product: Firefox for Android → Firefox Build System
Target Milestone: Firefox 69 → mozilla69
You need to log in before you can comment on or make changes to this bug.