Closed Bug 1487552 Opened 6 years ago Closed 5 years ago

Provide further steps to devs for xcode-select --install on Mojave (AKA 'inttypes.h' file not found)

Categories

(Firefox Build System :: General, defect, P2)

Product:
Firefox Build System
Component:
General
Platform:
Unspecified
macOS
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: alexical, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: in-triage)

Attachments

(2 files)

config.status
31.71 KB, text/plain
Details
config.status.diff
7.19 KB, patch
Details | Diff | Splinter Review
Just ran into this on the Mojave beta. Per the Xcode 10 beta release notes[1], xcode-select --install no longer writes headers to /usr/include, causing rust-bindgen to fail, so you need to manually run the installer at /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg to get them. We should account for this and either run it automatically or explain this to devs. [1]: https://download.developer.apple.com/Developer_Tools/Xcode_10_Beta/Xcode_10_Beta_Release_Notes.pdf
Keywords: in-triage
After reading bug 1366564 I feel like we ought to just make the build system pass the right SDK include paths down to bindgen by default.
(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #1) > After reading bug 1366564 I feel like we ought to just make the build system > pass the right SDK include paths down to bindgen by default. I tried this locally, and it was not too difficult to build FF without /usr/include headers. Adding -isysroot to CFLAGS/CXXFLAGS for bindgen, HostPrograms, and HostSimplePrograms, as well as fixing a couple configure issues, was enough to complete the build. Configure issues: - <unistd.h> was not detected by configure -> XPCShellImpl.cpp failed to compile. - <sys/mount.h> was not detected by configure -> nsLocalFileUnix.cpp failed to compile.
(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #1) > After reading bug 1366564 I feel like we ought to just make the build system > pass the right SDK include paths down to bindgen by default. I've explored a bit in bug 1509696. The main problem is that Homebrew clang still relies on /usr/include. We should pass MACOS_SDK_DIR with -isysroot to bindgen when it's specified, and probably use system clang to detect that otherwise. Actually I forgot why we need to use Homebrew clang in the first place... Can we just use the system clang instead?
(In reply to Xidorn Quan [:xidorn] UTC+11 from comment #4) > Actually I forgot why we need to use Homebrew clang in the first place... > Can we just use the system clang instead? I've been using system clang for quite a while.
(In reply to Jonathan Watt [:jwatt] from comment #5) > (In reply to Xidorn Quan [:xidorn] UTC+11 from comment #4) > > Actually I forgot why we need to use Homebrew clang in the first place... > > Can we just use the system clang instead? > > I've been using system clang for quite a while. For bindgen as well? I'd be interesting on how to do so... I don't recall all the details but it seems it's using homebrew clang by default...
Ah, no, sorry. I just mean I removed the setting of the CC and CXX vars from my .mozconfig
This bug appears to be pretty common and the build system ergonomics are pretty bad. Bumping the priority so it shows up on our radar. I'll bring this up with other build peers IRL today and hopefully we'll have a path forward by EOD.
Priority: -- → P2
From the linked PDF: The Command Line Tools package installs the macOS system headers inside the macOS SDK. Software that compiles with the installed tools will search for macOS headers at the standard include path: /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include. For legacy software that looks for the macOS headers in the base system under /usr/include, please install the package file located at: /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg. I agree with Ted in comment #1 that we should teach the build system to look for headers in SDK_ROOT/usr/include.
As I mentioned in bug 1509696 comment 9, I think the first step would be to move --with-macos-sdk from old-configure into moz.configure, so that we have control over MACOS_SDK_DIR there. We can then add auto-detection or so when that's not available, and also have it added to bindgen_cflags as -isysroot.
Also, it seems that /usr/bin/clang is just a thin wrapper of the true clang, but with -isysroot added to the argument. We may use that for detecting the sdk dir...
*bump* I just upgraded to Mojave and this is the first thing I hit when trying to build... Can we have a `./mach bootstrap` config thingy that fixes this? For example, when it says 'Looks like you have Homebrew installed...' to basically propose to upgrade known packages, or something or other. However, I think I've fixed it locally with `brew update && brew upgrade`.
Nope, that didn't work. Doing `brew uninstall --force llvm` also didn't change a thing.
(In reply to Mike de Boer [:mikedeboer] from comment #13) > Nope, that didn't work. Doing `brew uninstall --force llvm` also didn't > change a thing. Mike, are you perhaps also encountering Bug 1494022?
(In reply to Michael Froman [:mjf] from comment #14) > (In reply to Mike de Boer [:mikedeboer] from comment #13) > > Nope, that didn't work. Doing `brew uninstall --force llvm` also didn't > > change a thing. > > Mike, are you perhaps also encountering Bug 1494022? Oh dear, yes! I fixed the building part by applying https://bugzilla.mozilla.org/show_bug.cgi?id=1509696#c4, but now it's the all black indeed.

This bites people continuously, does this have any chance to get prioritized?

I also got compilation errors today after upgrading MacOS to Mojave (10.14). Finally built successfully after going through all the related bugs and comments...

  1. Download 10.13 sdk [1], and add ac_add_options --with-macos-sdk=path/to/MacOSX10.13.sdk to mozconfig. (from Bug 1494022)
  2. Run xcode-select --install and open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg (from this bug, comment 9) // To make sure we can get the llvm headers
  3. ./mach bootstrap // To make sure I didn't miss anything, and this forced me to upgrade brew
  4. ./mach clobber // For a clean build
  5. Clear the ccache and restart sccache
  6. ./mach build // Didn't use icecc

[1] https://www.avenard.org/files/MacOSX10.13.sdk.tbz

Hope everything is OK tomorrow.

I can confirm, since the build started failing for me again with Xidorn's bindgen.configure change applied, that Boris' steps are correct. Only change to step 2) is to run open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg (appending .pkg to it, since it's an installer).

Cheers, Boris. I really hope we can fix our build system soon, because this is causing a lot of frustration.

Changing description for better SEO.

Summary: Provide further steps to devs for xcode-select --install on Mojave → Provide further steps to devs for xcode-select --install on Mojave (AKA 'inttypes.h' file not found)

So, the above fixes the "inittypes.h" error, but then I get:

 0:06.18 toolkit/library/XUL
 0:12.11 ld: warning: could not create compact unwind for _ffi_call_unix64: does not use RBP or RSP based frame
 0:13.81 Undefined symbols for architecture x86_64:
 0:13.81   "_SecTrustEvaluateWithError", referenced from:
 0:13.83       GatherEnterpriseCertsMacOS(mozilla::Vector<EnterpriseCert, 0ul, mozilla::MallocAllocPolicy>&) in Unified_cpp_security_manager_ssl0.o
 0:14.31 ld: symbol(s) not found for architecture x86_64
 0:14.39 clang: error: linker command failed with exit code 1 (use -v to see invocation)
 0:14.39 make[4]: *** [XUL] Error 1
 0:14.40 make[3]: *** [toolkit/library/target] Error 2
 0:14.40 make[2]: *** [compile] Error 2
 0:14.40 make[1]: *** [default] Error 2
 0:14.40 make: *** [build] Error 2
 0:14.43 364 compiler warnings present.
 0:14.59 ccache (direct) hit rate: 0.0%; (preprocessed) hit rate: 0.0%; miss rate: 0.0%

Anyone seeing that too?

Depends on: 1495672
Blocks: 1495672
No longer depends on: 1495672

This is too easy for developers to hit when setting up their build env, or part way through if they've not used it for a while. Since this isn't a good experience, bumping the severity for this.

Mike could you look at this and determine how much work it would be to fix?

Flags: needinfo?(mshal)

After a recent update I had to downgrade from MacOSX10.13.sdk to MacOSX10.12.sdk due to:

 5:20.54 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/IOKit.framework/Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 5:20.54 #include <IOKit/IOReturn.h>
 5:20.54          ^~~~~~~~~~~~~~~~~~
 5:20.54 1 error generated.

Here's what worked for me:

  • Download 10.12.sdk from https://github.com/phracker/MacOSX-SDKs/.
  • sudo mv ~/Downloads/MacOSX10.12.sdk/ /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs
  • open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg and click through the UI
  • Add this to mozconfig:
    ac_add_options --with-macos-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/

(In reply to Brian Grinstead [:bgrins] from comment #25)

After a recent update I had to downgrade from MacOSX10.13.sdk to
MacOSX10.12.sdk due to:

 5:20.54
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/
Developer/SDKs/MacOSX10.13.sdk//System/Library/Frameworks/IOKit.framework/
Headers/IOTypes.h:38:10: fatal error: 'IOKit/IOReturn.h' file not found
 5:20.54 #include <IOKit/IOReturn.h>
 5:20.54          ^~~~~~~~~~~~~~~~~~
 5:20.54 1 error generated.

That sounds like bug 1544763.

I tried switching back to latest and greatest version of the SDK and it compiles fine again. No idea 🤷‍♂️

(In reply to Marcos Caceres [:marcosc] from comment #27)

I tried switching back to latest and greatest version of the SDK and it compiles fine again. No idea 🤷‍♂️

From https://bugzilla.mozilla.org/show_bug.cgi?id=1544418#c17 it sounds like always using -isysroot causes errors on different SDKs, and -isysroot was added because we started always checking the SDK version to try to make sure developers didn't waste time building with a known-bad SDK. We always use -isysroot in automation, but that is cross-compiled from Linux, and unfortunately we don't have any automated test coverage of native OSX builds anymore.

:miko, do you still have the patches to fix this from #c2? Are they sufficient to close out this bug?

Flags: needinfo?(mshal) → needinfo?(mikokm)

(In reply to Michael Shal [:mshal] from comment #29)

:miko, do you still have the patches to fix this from #c2? Are they sufficient to close out this bug?

Unfortunately, I do not have the patches anymore. It was a quick hack trying to get the build to work without installing system headers with 10.14 SDK, that eventually hit bug 1494022.

It was pretty trivial to fix the problems listed in comment 2 locally, but since I am not very familiar with the build system, I am not sure how portable my fixes were, for example, across different macOS SDK versions.

Flags: needinfo?(mikokm)

In bug 1495672 I've landed a configure check that checks for inttypes.h and prompts the developer to install the headers if necessarily.

I think ideally, we should have xcode-select --install and open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg as part of bootstrap (I don't think we do the xcode-select install currently) - that way new developers don't have to run configure twice to figure out the extra items they need.

After Mark's configure check was added, I tried things again and it all failed miserably (no worries, I managed to fix it after):

error: failed to run custom build command for `baldrdash v0.1.0 (/Users/mikedeboer/Projects/mozilla-central/js/src/wasm/cranelift)`
 9:35.64 process didn't exit successfully: `/Users/mikedeboer/Projects/mozilla-central/obj-x86_64-apple-darwin18.5.0/debug/build/baldrdash-5230329d7e773058/build-script-build` (exit code: 101)
 9:36.32 --- stdout
 9:36.32 cargo:rerun-if-changed=baldrapi.h
 9:36.32 cargo:rerun-if-changed=/Users/mikedeboer/Projects/mozilla-central/obj-x86_64-apple-darwin18.5.0/js/src/rust/extra-bindgen-flags
 9:36.32 --- stderr
 9:36.32 /usr/local/Cellar/llvm/8.0.0/lib/clang/8.0.0/include/inttypes.h:30:15: fatal error: 'inttypes.h' file not found
 9:36.32 /usr/local/Cellar/llvm/8.0.0/lib/clang/8.0.0/include/inttypes.h:30:15: fatal error: 'inttypes.h' file not found, err:

What happened here was that I did a brew uninstall --force llvm and a ./mach bootstrap, which re-installed llvm 8.
After I got the build failure above, I re-ran open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg and it magically worked! (Yes, I've ran the installer before on this machine. Numerous times, in fact.)
So another tip would be to just run the installer after everything else is done.

(In reply to Brian Grinstead [:bgrins] from comment #25)

Here's what worked for me:

To anyone else who's a bit concerned about the unverified nature of the phracker SDK repository: see bug 1522931 comment 27 for instructions on how to obtain a verified copy.

(In reply to Mark Banner (:standard8) from comment #31)

In bug 1495672 I've landed a configure check that checks for inttypes.h and prompts the developer to install the headers if necessarily.

I think ideally, we should have xcode-select --install and open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg as part of bootstrap (I don't think we do the xcode-select install currently) - that way new developers don't have to run configure twice to figure out the extra items they need.

I just updated to todays mozilla central, tried to build and ran into the build issue from bug 1495672. I guess that means the configure check from bug 1495672 is not working :-(

(In reply to Nils Ohlmeier [:drno] from comment #34)

(In reply to Mark Banner (:standard8) from comment #31)

In bug 1495672 I've landed a configure check that checks for inttypes.h and prompts the developer to install the headers if necessarily.

I think ideally, we should have xcode-select --install and open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg as part of bootstrap (I don't think we do the xcode-select install currently) - that way new developers don't have to run configure twice to figure out the extra items they need.

I just updated to todays mozilla central, tried to build and ran into the build issue from bug 1495672. I guess that means the configure check from bug 1495672 is not working :-(

Does reverting the patch in bug 1547847 help? I suppose it's possible target.kernel is never 'Darwin' and that's causing us to skip the check?

Flags: needinfo?(drno)

(In reply to Bobby Holley (:bholley) from comment #35)

Does reverting the patch in bug 1547847 help? I suppose it's possible target.kernel is never 'Darwin' and that's causing us to skip the check?

I did check host.kernel, target.kernel and version and they appear to have sensible values. As I installed the header package it now works and it turns out that appears to be impossible to delete a header file from /usr/include. So I can't repro any more until the next Xcode update comes along (unless someone has an idea how to repro).

Flags: needinfo?(drno)

I did manage to remove inttypes.h after disabling SPI. The fix from bug 1495672 detects properly a missing inttypes.h. My guess is that my system actually had inttypes.h present, and something else caused baldrdash compilation to fail.

I just pulled moz-central (df3eadfa74a8) and ran 'mach bootstrap' on macOS 10.14.4 this morning after doing system updates over the weekend and hit bug 1495672 again:
8:14.07 error: failed to run custom build command for baldrdash v0.1.0 (/Users/mfroman/mozilla/moz-central/js/src/wasm/cranelift)
8:14.07 process didn't exit successfully: /MozillaBuilds/obj/opt/release/build/baldrdash-a6bfb773b2da4010/build-script-build (exit code: 101)
8:14.07 --- stdout
8:14.07 cargo:rerun-if-changed=baldrapi.h
8:14.07 cargo:rerun-if-changed=/MozillaBuilds/obj/opt/js/src/rust/extra-bindgen-flags
8:14.07 --- stderr
8:14.07 /usr/local/Cellar/llvm/7.0.0_1/lib/clang/7.0.0/include/inttypes.h:30:15: fatal error: 'inttypes.h' file not found
8:14.07 /usr/local/Cellar/llvm/7.0.0_1/lib/clang/7.0.0/include/inttypes.h:30:15: fatal error: 'inttypes.h' file not found, err: true
8:14.07 thread 'main' panicked at 'Unable to generate baldrapi.h bindings: ()', src/libcore/result.rs:997

I've left my macOS system in this state in case there is info needed to help nail this down.

Can you try reverting the patch in bug 1547847 and checking whether configure detects the issue earlier?

Flags: needinfo?(mfroman)

Reversing the patch in Bug 1547847 and running 'mach build' makes it all the way through configure and dies with the same error.

For another datapoint, doing a 'mach clobber' didn't change the behavior, nor did trying a 'hg purge --all && mach clobber --full'.

Flags: needinfo?(mfroman)

(In reply to Michael Froman [:mjf] from comment #40)

Reversing the patch in Bug 1547847 and running 'mach build' makes it all the way through configure and dies with the same error.

OK, helpful to have that ruled out. Thanks.

Nathan, any thoughts on next step to try to figure out why the check from bug 1495672 isn't working?

Flags: needinfo?(nfroyd)

(In reply to Bobby Holley (:bholley) from comment #41)

(In reply to Michael Froman [:mjf] from comment #40)

Reversing the patch in Bug 1547847 and running 'mach build' makes it all the way through configure and dies with the same error.

OK, helpful to have that ruled out. Thanks.

Nathan, any thoughts on next step to try to figure out why the check from bug 1495672 isn't working?

Does $OBJDIR/config.log say anything about what the compiler invocation to check for <inttypes.h> was?

What does bindgen say about the compiler invocation to build the cranelift bindings?

What are the contents of $OBJDIR/js/src/rust/extra_bindgen_flags?

Flags: needinfo?(nfroyd) → needinfo?(mfroman)

(In reply to Nathan Froyd [:froydnj] from comment #42)

(In reply to Bobby Holley (:bholley) from comment #41)

(In reply to Michael Froman [:mjf] from comment #40)

Reversing the patch in Bug 1547847 and running 'mach build' makes it all the way through configure and dies with the same error.

OK, helpful to have that ruled out. Thanks.

Nathan, any thoughts on next step to try to figure out why the check from bug 1495672 isn't working?

Does $OBJDIR/config.log say anything about what the compiler invocation to check for <inttypes.h> was?

INFO: checking for inttypes.h...
DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.v32q2X.cpp with content:
DEBUG: | #include <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -isysroot /MozillaBuilds/MacOSX10.13.sdk -std=gnu++14 -c /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.v32q2X.cpp
INFO: yes

What does bindgen say about the compiler invocation to build the cranelift bindings?

I'm not sure I understand how to get the information you want here.

What are the contents of $OBJDIR/js/src/rust/extra_bindgen_flags?

-x c++ -fno-sized-deallocation -DTRACING=1 -DIMPL_LIBXUL -DMOZILLA_INTERNAL_API -DRUST_BINDGEN -DOS_POSIX=1 -DOS_MACOSX=1 -stdlib=libc++ -I/MozillaBuilds/obj/opt/dist/include/nspr

Flags: needinfo?(mfroman)

(In reply to Michael Froman [:mjf] from comment #43)

INFO: checking for inttypes.h...
DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.v32q2X.cpp with content:
DEBUG: | #include <inttypes.h>

That's #include <inttypes.h>. Maybe we can't use #include_next to test this? Though I'm sure it was working locally when I tried, but maybe I missed something.

(In reply to Michael Froman [:mjf] from comment #43)

(In reply to Nathan Froyd [:froydnj] from comment #42)

(In reply to Bobby Holley (:bholley) from comment #41)

(In reply to Michael Froman [:mjf] from comment #40)

Reversing the patch in Bug 1547847 and running 'mach build' makes it all the way through configure and dies with the same error.

OK, helpful to have that ruled out. Thanks.

Nathan, any thoughts on next step to try to figure out why the check from bug 1495672 isn't working?

Does $OBJDIR/config.log say anything about what the compiler invocation to check for <inttypes.h> was?

INFO: checking for inttypes.h...
DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.v32q2X.cpp with content:
DEBUG: | #include <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -isysroot /MozillaBuilds/MacOSX10.13.sdk -std=gnu++14 -c /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.v32q2X.cpp
INFO: yes

I meant to ask about the compiler invocation to check #include_next <inttypes.h>, but this is helpful too! Can you check the log for the #include_next <inttypes.h> check as well?

What does bindgen say about the compiler invocation to build the cranelift bindings?

I'm not sure I understand how to get the information you want here.

Ah, stylo will spit out information about what flags were used when invoking clang...I see the JS side of things doesn't do that. I'll file a bug to fix that.

What are the contents of $OBJDIR/js/src/rust/extra_bindgen_flags?

-x c++ -fno-sized-deallocation -DTRACING=1 -DIMPL_LIBXUL -DMOZILLA_INTERNAL_API -DRUST_BINDGEN -DOS_POSIX=1 -DOS_MACOSX=1 -stdlib=libc++ -I/MozillaBuilds/obj/opt/dist/include/nspr

OK, so there's no -isysroot flag being passed here, which is probably bad. What does your mozconfig look like?

Flags: needinfo?(mfroman)

(In reply to Nathan Froyd [:froydnj] from comment #45)

I meant to ask about the compiler invocation to check #include_next <inttypes.h>, but this is helpful too! Can you check the log for the #include_next <inttypes.h> check as well?

js/src> DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.Ibvf7q.cpp with content:
js/src> DEBUG: | #include_next <inttypes.h>
js/src> DEBUG: | int
js/src> DEBUG: | main(void)
js/src> DEBUG: | {
js/src> DEBUG: |
js/src> DEBUG: | ;
js/src> DEBUG: | return 0;
js/src> DEBUG: | }
js/src> DEBUG: Executing: `/usr/local/bin/ccache /usr/bin/clang++ -isysroot /MozillaBuilds/MacOSX10.13.sdk -std=gnu++14 /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/c
js/src> INFO: checking for 64-bit OS...

What does bindgen say about the compiler invocation to build the cranelift bindings?

I'm not sure I understand how to get the information you want here.

Ah, stylo will spit out information about what flags were used when invoking clang...I see the JS side of things doesn't do that. I'll file a bug to fix that.

What are the contents of $OBJDIR/js/src/rust/extra_bindgen_flags?

-x c++ -fno-sized-deallocation -DTRACING=1 -DIMPL_LIBXUL -DMOZILLA_INTERNAL_API -DRUST_BINDGEN -DOS_POSIX=1 -DOS_MACOSX=1 -stdlib=libc++ -I/MozillaBuilds/obj/opt/dist/include/nspr

OK, so there's no -isysroot flag being passed here, which is probably bad. What does your mozconfig look like?

#ac_add_options --enable-debug
#ac_add_options --disable-optimize
#ac_add_options --enable-warnings-as-errors
#ac_add_options --enable-debug-symbols

export SYS_NAME=`uname`
export CONFIG_NAME=`currentConfig`
if [ "x$SYS_NAME" = "xDarwin" ]; then
  export CC=clang
  export CXX=clang++
  #ICECC on OS X - from instructions here: https://github.com/jyavenard/mozilla-icecream
#  export CC="/usr/local/opt/llvm/bin/clang --target=x86_64-apple-darwin16.0.0 -mmacosx-version-min=10.11"
#  export CXX="/usr/local/opt/llvm/bin/clang++ --target=x86_64-apple-darwin16.0.0 -mmacosx-version-min=10.11"

#  ac_add_options --with-compiler-wrapper="/usr/local/bin/icecc"
#  mk_add_options 'export ICECC_VERSION=x86_64:/Users/mfroman/clang-icecc/clang-5.0.0-x86_64.tar.gz,Darwin17_x86_64:/Users/mfroman/clang-icecc/clang-5.0.0-Darwin17_x86_64.tar.gz'

  # to fix bug 1494022 black window travesty
  ac_add_options --with-macos-sdk=/MozillaBuilds/MacOSX10.13.sdk

  # core count of moz box, moz-build VM (on OSX desktop), and local
  export BUILD_JOBS=8
  export BUILD_OBJ_BASEDIR=/MozillaBuilds

else
  #ICECC on Linux
  export CC=$HOME/.mozbuild/clang/bin/clang
  export CXX=$HOME/.mozbuild/clang/bin/clang++

  # core count of moz box (not using icecc here yet)
  export BUILD_JOBS=28
  export BUILD_OBJ_BASEDIR=/home/mfroman/mozilla

fi

# Turn off rust optimizations to improve build times
export RUSTC_OPT_LEVEL=0
mk_add_options "export RUSTC_WRAPPER=sccache"
mk_add_options MOZ_OBJDIR=$BUILD_OBJ_BASEDIR/obj/$CONFIG_NAME
#mk_add_options 'export CCACHE_PREFIX=icecc'
mk_add_options MOZ_MAKE_FLAGS=-j$BUILD_JOBS
mk_add_options AUTOCLOBBER=1

export MOZ_WEBRTC_TESTS=1
export MOZ_WEBRTC_LEAKING_TESTS=1
export MOZ_CRASHREPORTER=1

# telemetry exports - adds lots of build time
#export MOZ_TELEMETRY_REPORTING=1
#export MOZILLA_OFFICIAL=1

# ccache stuff
ac_add_options --with-ccache
Flags: needinfo?(mfroman)

Sorry about the formatting there. I wasn't expecting that... I fixed the markdown, so it should look correct now.

Depends on: 1549794

Ah, thanks. Can you try building with the patch in bug 1549794, and see if that makes things any better?

Flags: needinfo?(mfroman)

./mach clobber --full && hg purge && ./mach build with the patch from bug 1549794 yields the same result.

DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.dXYwG8.cpp with content:
DEBUG: | #include_next <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -std=gnu++14 /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.dXYwG8.cpp
INFO: checking for 64-bit OS...

...

INFO: checking for inttypes.h...
DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.bbS2f8.cpp with content:
DEBUG: | #include <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -std=gnu++14 -c /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.bbS2f8.cpp
INFO: yes

extra-bindgen-flags:

-x c++ -fno-sized-deallocation -DTRACING=1 -DIMPL_LIBXUL -DMOZILLA_INTERNAL_API -DRUST_BINDGEN -DOS_POSIX=1 -DOS_MACOSX=1 -stdlib=libc++ -I/MozillaBuilds/obj/opt/dist/include/nspr

Flags: needinfo?(mfroman)
Flags: needinfo?(nfroyd)

(In reply to Michael Froman [:mjf] from comment #49)

./mach clobber --full && hg purge && ./mach build with the patch from bug 1549794 yields the same result.

DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.dXYwG8.cpp with content:
DEBUG: | #include_next <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -std=gnu++14 /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.dXYwG8.cpp

Wait, what? Where did the -isysroot from comment 43 and 46 go? Did the mozconfig change?

Flags: needinfo?(nfroyd)

Same mozconfig. Hmm... There may have been something left over from reversing the patch in Bug 1547847 that I did back in comment 39/40. Let me do that again, and I'll see if the -isysroot is back. Sorry.

Nope - that wasn't it...

DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.zJ4Rcz.cpp with content:
DEBUG: | #include_next <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -std=gnu++14 /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.zJ4Rcz.cpp

INFO: checking for inttypes.h...
DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.MUDZ7M.cpp with content:
DEBUG: | #include <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -std=gnu++14 -c /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.MUDZ7M.cpp
INFO: yes

Maybe something different when I did a 'hg purge --all' vs 'hg purge'?

(In reply to Michael Froman [:mjf] from comment #51)

Same mozconfig. Hmm... There may have been something left over from reversing the patch in Bug 1547847 that I did back in comment 39/40. Let me do that again, and I'll see if the -isysroot is back. Sorry.

You are using clang from a brew install, and not the latest version.

Try uninstalling llvm brew package, or upgrading brew llvm.

Just lost a whole heap of time trying to compile on Mojave. Running:

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

finally got me compiling, which seems to indicate that baldrapi doesn't use the --with-macos-sdk from my mozconfig.

(In reply to Michael Froman [:mjf] from comment #53)

Maybe something different when I did a 'hg purge --all' vs 'hg purge'?

Ok - sorry, I did a purge, clobber, cleared ccache, etc. Rebuilt, and now the -isysroot is back where you were expecting it. I must have mistyped something along the way last time.

DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.pUOH9a.cpp with content:
DEBUG: | #include_next <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -isysroot /MozillaBuilds/MacOSX10.13.sdk -std=gnu++14 /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.pUOH9a.cpp
INFO: checking for 64-bit OS...

...

INFO: checking for inttypes.h...
DEBUG: Creating /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.e6MOyA.cpp with content:
DEBUG: | #include <inttypes.h>
DEBUG: | int
DEBUG: | main(void)
DEBUG: | {
DEBUG: |
DEBUG: | ;
DEBUG: | return 0;
DEBUG: | }
DEBUG: Executing: /usr/local/bin/ccache /usr/bin/clang++ -isysroot /MozillaBuilds/MacOSX10.13.sdk -std=gnu++14 -c /var/folders/5p/x2tk851538z3dvbf5wg0x_sc0000gn/T/conftest.e6MOyA.cpp
INFO: yes

Flags: needinfo?(nfroyd)

(In reply to Jean-Yves Avenard [:jya] from comment #54)

(In reply to Michael Froman [:mjf] from comment #51)

Same mozconfig. Hmm... There may have been something left over from reversing the patch in Bug 1547847 that I did back in comment 39/40. Let me do that again, and I'll see if the -isysroot is back. Sorry.

You are using clang from a brew install, and not the latest version.

Try uninstalling llvm brew package, or upgrading brew llvm.
brew outdated showed:
llvm (7.0.0_1) < 8.0.0

I did a brew upgrade llvm, and it did not change the behavior.

To be clear, comment 57 is with the patch from bug 1549794 applied, yes? If that's the case, can you attach your config.status from the relevant objdir? If it's not the case, can you retest with that patch applied, and attach config.status if things still don't appear to work?

Flags: needinfo?(nfroyd)

Correct, that was with the patch from bug 1549794 applied.

Attached file config.status —

OK, this is kind of bizarre. I can see from config.status that we have -isysroot in our bindgen flags, so the patch worked:

'BINDGEN_SYSTEM_FLAGS': '-x c++ -fno-sized-deallocation -DTRACING=1 -DIMPL_LIBXUL -DMOZILLA_INTERNAL_API -DRUST_BINDGEN -DOS_POSIX=1 -DOS_MACOSX=1 -stdlib=libc++ -isysroot /MozillaBuilds/MacOSX10.13.sdk -I/MozillaBuilds/obj/opt/dist/include/nspr  -I/MozillaBuilds/obj/opt/dist/include/nss  -I/MozillaBuilds/obj/opt/dist/include/cairo'

We're using /usr/bin/clang for our C/C++ compiler, and that compiler is getting the necessary -isysroot flags. But we're using /usr/local/Cellar/llvm/8.0.0/bin/clang for our bindgen clang, and either that compiler is getting the necessary -isysroot flags and ignoring them (?), it's getting the necessary flags, but that's not enough for some reason, or it's not getting the necessary flags. mjf, can you confirm what the contents of $OBJDIR/js/src/rust/extra-bindgen-flags are with the patch from bug 1549794 applied?

Flags: needinfo?(mfroman)

Contents of $OBJDIR/js/src/rust/extra-bindgen-flags with patch from bug 1549794 applied:

-x c++ -fno-sized-deallocation -DTRACING=1 -DIMPL_LIBXUL -DMOZILLA_INTERNAL_API -DRUST_BINDGEN -DOS_POSIX=1 -DOS_MACOSX=1 -stdlib=libc++ -I/MozillaBuilds/obj/opt/dist/include/nspr

Flags: needinfo?(mfroman)

I ran into the additional aspect that I had to add to mozconfig:

export LIBCLANG_PATH=/usr/local/Cellar/llvm/8.0.0/lib/

otherwise baldrdash failed to find libclang.dylib

(In reply to Shane Caraveo (:mixedpuppy) from comment #63)

I ran into the additional aspect that I had to add to mozconfig:

export LIBCLANG_PATH=/usr/local/Cellar/llvm/8.0.0/lib/

otherwise baldrdash failed to find libclang.dylib

What did your mozconfig look like prior to this? You shouldn't have to specify this manually; we should be able to derive it from the llvm-config in your PATH.

Flags: needinfo?(mixedpuppy)

(In reply to Nathan Froyd [:froydnj] from comment #64)

What did your mozconfig look like prior to this?

I didn't have one. I usually only use one if I'm using artifact builds or need debug.

Flags: needinfo?(mixedpuppy)

(In reply to Shane Caraveo (:mixedpuppy) from comment #65)

(In reply to Nathan Froyd [:froydnj] from comment #64)

What did your mozconfig look like prior to this?

I didn't have one. I usually only use one if I'm using artifact builds or need debug.

That's...peculiar. If you configure into a different objdir (or use the default, if you don't mind losing your build), but don't include the LIBCLANG_PATH export, can you attach your $objdir/config.status?

Flags: needinfo?(mixedpuppy)

Well, that's weird. When I was trying to figure out the inttypes.h problem I nuked my obj dir and still hit the clang problem. Now all I did was change the obj dir in mozconfig (not setting the clang path) and it's building without problem.

Here's a diff between the regular obj dir I had problems with, and the new on that is building, I don't see anything obvious.

Assignee: nobody → mixedpuppy
Flags: needinfo?(mixedpuppy)
Assignee: mixedpuppy → nobody

kmoir, can you assign this?

Flags: needinfo?(kmoir)

I believe this will be fixed by bug 1526857, which is now at the top of my todo list.

It looks like glandium is looking at this as part of bug 1526857.

Depends on: 1526857
Flags: needinfo?(kmoir)

Mike now that bug 1526857 is fixed, should we back out my patch from bug 1495672 (which probably wasn't completely working anyway), and then call this bug fixed?

Flags: needinfo?(mh+mozilla)

Maybe?

Flags: needinfo?(mh+mozilla)

We haven't seen this with new machines attempting to build recently, please file new bugs for issues as you see them.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: