1331957 - update clang static analysis builds to 3.9

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Nathan Froyd [:froydnj]]

We updated the linux one in bug 1331012, we should do the same for Mac as per bug 1331012 comment 6.

If we can cross-compile clang on infra, I'll just take responsibility for making that work, since I need to make Mac builds for the stylo work and ensuring that mach bootstrap can download things from infra.  Sound good, Ehsan?

Comment 1

[(no longer active)]

I started working on this yesterday, so far the build is red: <https://treeherder.mozilla.org/#/jobs?repo=try&revision=e07af4b5a6ef32e750c64cc4da3bc87dd505d7a8>.  I'm trying to run it locally to see what's going on...  Should I stop?

Comment 2

[Nathan Froyd [:froydnj]]

(In reply to :Ehsan Akhgari from comment #1)
> I started working on this yesterday, so far the build is red:
> <https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=e07af4b5a6ef32e750c64cc4da3bc87dd505d7a8>.  I'm
> trying to run it locally to see what's going on...  Should I stop?

Oh, didn't realize you had started already!  I guess you can continue making this work, then.

Comment 3

[(no longer active)]

It turns out that the reason the build was busted is https://llvm.org/bugs/show_bug.cgi?id=28831.  The fix didn't make it to 3.9, so we need to cherry-pick it.

Comment 4

[(no longer active)]

Another thing to note is that on the infrastructure, we obviously can't build a 3 stage compiler since we can't run the stage1 compiler.

Comment 5

[(no longer active)]

I ran into an issue with the cross-compiled clang.  It seems that system ld cannot open the clang runtime library generated in the build:

$ /tmp/clang/clang/bin/clang test.c -v
clang version 3.9.0 (tags/RELEASE_390/final) (llvm/tags/RELEASE_390/final 290136)
Target: x86_64-apple-darwin16.3.0
Thread model: posix
InstalledDir: /tmp/clang/clang/bin
 "/private/tmp/clang/clang/bin/clang-3.9" -cc1 -triple x86_64-apple-macosx10.12.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name test.c -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -masm-verbose -munwind-tables -target-cpu core2 -target-linker-version 3.8.0 -v -dwarf-column-info -debugger-tuning=lldb -resource-dir /private/tmp/clang/clang/bin/../lib/clang/3.9.0 -fdebug-compilation-dir /private/tmp/clang/cctools-port/cctools -ferror-limit 19 -fmessage-length 180 -stack-protector 1 -fblocks -fobjc-runtime=macosx-10.12.0 -fencode-extended-block-signature -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o /var/folders/ry/m4kkhrl17c76cqyfd5qv5sph0000gn/T/test-d3ce84.o -x c test.c
clang -cc1 version 3.9.0 based upon LLVM 3.9.0 default target x86_64-apple-darwin16.3.0
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /private/tmp/clang/clang/bin/../lib/clang/3.9.0/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
 "/usr/bin/ld" -dynamic -arch x86_64 -macosx_version_min 10.12.0 -o a.out /var/folders/ry/m4kkhrl17c76cqyfd5qv5sph0000gn/T/test-d3ce84.o -lSystem /private/tmp/clang/clang/bin/../lib/clang/3.9.0/lib/darwin/libclang_rt.osx.a
ld: file is universal (1 slices) but does not contain a(n) x86_64 slice: /private/tmp/clang/clang/bin/../lib/clang/3.9.0/lib/darwin/libclang_rt.osx.a file '/private/tmp/clang/clang/bin/../lib/clang/3.9.0/lib/darwin/libclang_rt.osx.a' for architecture x86_64
clang-3.9: error: linker command failed with exit code 1 (use -v to see invocation)
$ file /private/tmp/clang/clang/bin/../lib/clang/3.9.0/lib/darwin/libclang_rt.osx.a
/private/tmp/clang/clang/bin/../lib/clang/3.9.0/lib/darwin/libclang_rt.osx.a: Mach-O universal binary with 1 architecture: [x86_64: current ar archive random library]
/private/tmp/clang/clang/bin/../lib/clang/3.9.0/lib/darwin/libclang_rt.osx.a (for architecture x86_64):	current ar archive random library

Nathan, have you ever encountered this?

Comment 6

[Nathan Froyd [:froydnj]]

I've seen things like bug 921040 comment 71, which is vaguely similar.  Are you sure that Apple's cctools (ar and ranlib are the important ones particularly, IIRC) are being used in the cross compile, rather than something cobbled together from binutils?  I wrote a patch for binutils:

https://sourceware.org/ml/binutils/2013-10/msg00360.html

but AFAICT, it didn't make it upstream.  (I also wrote one to make static libraries smaller:

https://sourceware.org/ml/binutils/2013-10/msg00359.html

which might also matter if we were using cross-compiled libc++.a files.)

Comment 7

[(no longer active)]

I decided to just debug what's happening inside ld64, and it turns out that the libclang_rt.osx.a archive has a chkstk.S.o file inside it that is an ELF file, not a MachO file, which is why ld64 barfs.

Comment 8

[Nathan Froyd [:froydnj]]

That would certainly cause problems!  Turn off stack checking, then?

Comment 9

[(no longer active)]

No, the problem is we're not passing -target in CMAKE_ASM_FLAGS, so none of the .S files are cross-compiled.  :(

Comment 10

[(no longer active)]

(In reply to :Ehsan Akhgari from comment #9)
> No, the problem is we're not passing -target in CMAKE_ASM_FLAGS, so none of
> the .S files are cross-compiled.  :(

Unfortunately merely passing CMAKE_ASM_FLAGS isn't enough because of <https://github.com/llvm-mirror/compiler-rt/blob/212ca9be08d3d542f8929754e651d42a6138fc41/cmake/Modules/CompilerRTDarwinUtils.cmake#L269>, so I have to patch compiler-rt!

Comment 11

[(no longer active)]

Continuing the saga here, the next issue that I hit was that some fix required by our clang plugin isn't backported on 3.9, so we need to cherry-pick that one as well.

When I did that, I noticed that in the static analysis builds, the clang cross-compiled on Linux can't load the plugin we build.  After some investigation it became clear that the reason is that the linker in the cctools version that we use for our Mac cross compile builds is too old, and it doesn't support the --export_dynamic option which must be used to tell the linker to include all symbols in the produced binary so that the plugin, once loaded, can link against those symbols.

I found two build scripts lying around, http://searchfox.org/mozilla-central/source/build/macosx/build-cctools.sh which is hard to say how it must be invoked, and http://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/build-cctools.sh which uses crosstools-ng, and as far as I can tell is probably what produced the cctools package we have in tooltool.  I tried to make this script build the latest cctools package available from opensource.apple.com but it seems that all of the newer packages are distributed with a build system configured for OSX, so it's impossible to build them on Linux.  After I learned that, I looked for alternatives, and I found https://github.com/tpoechtrager/cctools-port which can be successfully built to get a much newer linker, version 264.3.102.  With that, and changing the -mlinker-version flags we pass to the compiler from 136 to 137 (which is the minimum linker version at which clang, once you pass -rdynamic to it, translates that to --export_dynamic when constructing the linker's command line argument.  A cross compiled clang with this newer cctools toolchain can successfully load the clang plugin we build.

I think this was the last issue to fix.  Once I get a full green run on the try server I'll post my patches here.

Comment 12

[(no longer active)]

Ted, do you think we should try to upgrade our Mac cross compile builds to use this newer cctools package I have made?

Comment 13

[(no longer active)]

... and while this compiler wokrs for me locally, it doesn't work on our builders: <https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-ce11e737bc50ebbb014b6247c7e83bf9a1450dd0/try-macosx64-st-an-debug/try-macosx64-st-an-debug-bm83-try1-build17339.txt.gz>.  Since the log has no useful information in it, I'm going to need to request a slave to see what's going on.

Comment 14

[(not currently active) Ted Mielczarek]

(In reply to :Ehsan Akhgari from comment #12)
> Ted, do you think we should try to upgrade our Mac cross compile builds to
> use this newer cctools package I have made?

That would be great, actually, because the outdated linker we're using is also blocking updating our builds to a newer SDK (bug 1324892), which ought to fix a crash users are seeing (bug 1320048).

Comment 15

[(no longer active)]

I finally understood what's going wrong on the build machines!

libclang-plugin.dylib would fail to load with error "Symbol not found: __ZNKSt11logic_error4whatEv\n  Referenced from: /Users/cltbld/build/src/clang/bin/../lib/libc++.1.dylib\n  Expected in: /usr/lib/libc++abi.dylib".  This was a mystery because __ZNKSt11logic_error4whatEv was defined in /usr/lib/libc++abi.dylib on my system, but not on the builder.  There, it was defined in /usr/lib/libc++.dylib.  It turns out that there was an -L../../clang/lib in LLVM_LDFLAGS, and since clang 3.9 was built with libc++, there is a libc++ in clang/lib now, so that would be picked up first by the linker while resolving -lc++, so we would link against @rpath/libc++.1.dylib and @rpath/libc++abi.1.dylib instead of /usr/lib/libc++.1.dylib and /usr/lib/libc++abi.1.dylib.  But the libc++.1.dylib library in clang/lib links against /usr/lib/libc++abi.1.dylib, and it expects the ABI library to correspond to this newer libc++ version, but on the builder machines we would try to resolve the newer symbol against an older library.

The solution is to remove the -L argument from LLVM_LDFLAGS.

Comment 16

[(no longer active)]

Having fixed the above issue, I am not hitting an error when running ranlib to create libhostbz2.a, like this:

https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-3890c7f4beb6b860677d8e836b51ab598af1de95/try-macosx64-st-an-debug/try-macosx64-st-an-debug-bm83-try1-build17564.txt.gz
18:57:45     INFO -  Executing: ar crs libhostbz2.a host_blocksort.o host_bzlib.o host_compress.o host_crctable.o host_decompress.o host_huffman.o host_randtable.o
18:57:45     INFO -  /usr/bin/ranlib: object: libhostbz2.a(host_bzlib.o) malformed object (unknown load command 2)
18:57:45     INFO -  ar: internal ranlib command failed
18:57:45     INFO -  make[5]: *** [libhostbz2.a] Error 1
18:57:45     INFO -  make[5]: *** Deleting file `libhostbz2.a'
18:57:45     INFO -  make[4]: *** [modules/libbz2/src/host] Error 2

So far I have verified that some of these object files trigger this.  For example:

$ ar crs libhostbz2.a host_decompress.o
/usr/bin/ranlib: object: libhostbz2.a(host_decompress.o) malformed object (unknown load command 2)
ar: internal ranlib command failed

However I haven't managed to figure out why this is happening and what's special about this object file.  The ar and ranlib binaries shipping with the latest OSX on my local system don't trigger this issue.

Nathan, do you have any suggestions as to what may be the cause?

Comment 17

[(no longer active)]

I realized that load command means something in the MachO format!  After some investigation with, here is the difference:

$ otool -l host_decomress.o # on the slave
...
Load command 2
      cmd ?(0x00000029) Unknown load command
  cmdsize 16
00004a68 00000008

# otool -l host_decompress.o # locally
...
Load command 2
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 19048
 datasize 8

Looking at the LLVM code that emits this <https://github.com/llvm-mirror/llvm/blob/374362d92036c3f80e17bcc5051b5451498b350a/lib/MC/MachObjectWriter.cpp#L843>, it can't be disabled based on a flag, so the only way out of this is to upgrade the toolchain.  :/

Comment 18

[Nathan Froyd [:froydnj]]

Looks like you've figured this out.  We did use an upgraded ld64 for our Mac builds for a while, so I don't see any reason why we couldn't use an upgraded otool/ranlib/ar from a newer cctools as well..

Comment 19

[(no longer active)]

(In reply to Nathan Froyd [:froydnj] from comment #18)
> Looks like you've figured this out.  We did use an upgraded ld64 for our Mac
> builds for a while, so I don't see any reason why we couldn't use an
> upgraded otool/ranlib/ar from a newer cctools as well..

Except that building these tools from the source packages that Apple provides is rocket science.  :(

I'm now trying to see if it's sufficient to just zip up a bunch of these tools from my Xcode destination and upload to tooltool.

Comment 20

[Nathan Froyd [:froydnj]]

(In reply to :Ehsan Akhgari from comment #19)
> (In reply to Nathan Froyd [:froydnj] from comment #18)
> > Looks like you've figured this out.  We did use an upgraded ld64 for our Mac
> > builds for a while, so I don't see any reason why we couldn't use an
> > upgraded otool/ranlib/ar from a newer cctools as well..
> 
> Except that building these tools from the source packages that Apple
> provides is rocket science.  :(

That's why you don't build them from the source packages that Apple provides.  You build them from the ports that Tor or related folks have made, e.g. bug 1289847 comment 13.

> I'm now trying to see if it's sufficient to just zip up a bunch of these
> tools from my Xcode destination and upload to tooltool.

That might work, too!

Comment 21

[(no longer active)]

(In reply to Nathan Froyd [:froydnj] from comment #20)
> (In reply to :Ehsan Akhgari from comment #19)
> > (In reply to Nathan Froyd [:froydnj] from comment #18)
> > > Looks like you've figured this out.  We did use an upgraded ld64 for our Mac
> > > builds for a while, so I don't see any reason why we couldn't use an
> > > upgraded otool/ranlib/ar from a newer cctools as well..
> > 
> > Except that building these tools from the source packages that Apple
> > provides is rocket science.  :(
> 
> That's why you don't build them from the source packages that Apple
> provides.  You build them from the ports that Tor or related folks have
> made, e.g. bug 1289847 comment 13.

I already did this in comment 11, but the toolchain built there is for ELF.  I added a new job to build it for OS X as well.

> > I'm now trying to see if it's sufficient to just zip up a bunch of these
> > tools from my Xcode destination and upload to tooltool.
> 
> That might work, too!

Unfortunately it ended up not working.  ar started to SIGILL in the middle of the build, possibly because the binaries are targeting newer processor features.  Using the cctools I have built seems to work.

The next thing I hit is that the fix that made bug 1291397 go away didn't land on the 3.9 time frame, so I have to backport it.

Comment 22

[(no longer active)]

This are finally green! https://treeherder.mozilla.org/#/jobs?repo=try&revision=20fd9b91037835ebf708095bc03d5cad6019c737

Comment 23

[(no longer active)]

Attachment: Part 1: Update the clang OSX build config file for cross-compile builds on the infrastructure

Comment 24

[(no longer active)]

Attachment: Part 2: Add cross compile jobs for clang 3.9 OSX builds

Comment 25

[(no longer active)]

Attachment: Part 3: Delete some unused patches

Comment 26

[(no longer active)]

Attachment: Part 4: Allow specifying a custom assembler and pass the right flags to it when cross-compiling

Comment 27

[(no longer active)]

Attachment: Part 5: Add `-target x86_64-apple-darwin10' to the compiler-rt overridden CFLAGS

Comment 28

[(no longer active)]

Attachment: Part 6: Reapply LLVM r277806 since our clang-plugin depends on it

Comment 29

[(no longer active)]

Attachment: Part 7: Reapply LLVM r285657 to work around bug 1291397 on clang 3.9

Comment 30

[(no longer active)]

Attachment: Part 8: Build cctools-port in the infrastructure

Comment 31

[(no longer active)]

Attachment: Part 9: Upgrade cctools used for building clang on OS X for ld 264.3.102

Comment 32

[(no longer active)]

Attachment: Part 10: Upgrade cctools used for building Firefox on OS X debug static analysis for support for LC_DATA_IN_CODE sections

Comment 33

[(no longer active)]

Attachment: Part 11: Specify the path to the cross-compiler toolchain libtool for OS X static analysis builds

Comment 34

[(no longer active)]

Attachment: Part 12: Remove -Lclang/lib from the LDFLAGS used to build the clang plugin

This ensures that we correctly link against /usr/lib/libc++.1.dylib.
With this flag left in LDFLAGS, the linker would find libc++.1.dylib in
clang/lib, which would cause us to link against @rpath/libc++.1.dylib.

Comment 35

[(no longer active)]

Attachment: Part 13: Upgrade OSX static anlaysis builds to clang 3.9

Comment 36

[Nathan Froyd [:froydnj]]

(In reply to :Ehsan Akhgari from comment #22)
> This are finally green!

\o/

Just to confirm: the green jobs are just for building the cross-compiler and not for using the new cross-compiler in whatever jobs it's going to be used for?

Comment 37

[(no longer active)]

(In reply to Nathan Froyd [:froydnj] from comment #36)
> (In reply to :Ehsan Akhgari from comment #22)
> > This are finally green!
> 
> \o/
> 
> Just to confirm: the green jobs are just for building the cross-compiler and
> not for using the new cross-compiler in whatever jobs it's going to be used
> for?

It is for both.  The last patch in the patch set switches macosx64-st-an debug jobs to clang 3.9.

Comment 38

[(no longer active)]

Well, it is actually for three things: building the right versions of cctools-port for Linux and OSX, building clang 3.9 for OSX, and switching our OSX static analysis build to 3.9.

Comment 39

[Nathan Froyd [:froydnj]]

(In reply to :Ehsan Akhgari from comment #37)
> It is for both.  The last patch in the patch set switches macosx64-st-an
> debug jobs to clang 3.9.

\o/

Comment 40

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8832929 [details] [diff] [review]
Part 8: Build cctools-port in the infrastructure

Review of attachment 8832929 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/scripts/misc/build-cctools-port-macosx.sh
@@ +1,4 @@
> +#!/bin/bash
> +set -x -e -v
> +
> +# This script is for building cctools (Apple's binutils) for Linux using

..for OS X?

Comment 41

[Nathan Froyd [:froydnj]]

Comment on attachment 8832925 [details] [diff] [review]
Part 4: Allow specifying a custom assembler and pass the right flags to it when cross-compiling

Review of attachment 8832925 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/build-clang/clang-static-analysis-win64.json
@@ +10,5 @@
>      "libcxx_repo": "https://llvm.org/svn/llvm-project/libcxx/trunk",
>      "python_path": "c:/mozilla-build/python/python.exe",
>      "cc": "cl.exe",
>      "cxx": "cl.exe",
> +    "ml": "ml64.exe",

Are we not updating the clang-static-analysis-win32.json file as well?

Comment 42

[Nathan Froyd [:froydnj]]

Comment on attachment 8832929 [details] [diff] [review]
Part 8: Build cctools-port in the infrastructure

Review of attachment 8832929 [details] [diff] [review]:
-----------------------------------------------------------------

What Dustin said.

Comment 43

[Nathan Froyd [:froydnj]]

Comment on attachment 8832931 [details] [diff] [review]
Part 10: Upgrade cctools used for building Firefox on OS X debug static analysis for support for LC_DATA_IN_CODE sections

Review of attachment 8832931 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/mozconfigs/macosx64/debug-static-analysis
@@ +3,5 @@
>  MOZ_AUTOMATION_L10N_CHECK=0
>  
> +# The toolchain installed on our OSX 10.7 build machines is too old to support
> +# MachO LC_DATA_IN_CODE load command, which newer LLVM generates, so we need to
> +# use a newer toolchain that we build.

We don't need these assignments for the opt-static-analysis mozconfig?  Why not?

Comment 44

[Nathan Froyd [:froydnj]]

Comment on attachment 8832932 [details] [diff] [review]
Part 11: Specify the path to the cross-compiler toolchain libtool for OS X static analysis builds

Review of attachment 8832932 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with the below change.

::: build/build-clang/build-clang.py
@@ +472,5 @@
>      ar = get_tool(config, "lib" if is_windows() else "ar")
>      ranlib = None if is_windows() else get_tool(config, "ranlib")
> +    libtool = None
> +    if "ranlib" in config:
> +        libtool = get_tool(config, "libtool")

Did you really mean to check for "ranlib" and then get the "libtool" tool?  If so, you should change the docs accordingly.  If not, then you should fix the condition here.

Comment 45

[Nathan Froyd [:froydnj]]

Comment on attachment 8832933 [details] [diff] [review]
Part 12: Remove -Lclang/lib from the LDFLAGS used to build the clang plugin

Review of attachment 8832933 [details] [diff] [review]:
-----------------------------------------------------------------

I think we can do better than some horrible sed expression, see below.

::: build/autoconf/clang-plugin.m4
@@ +55,5 @@
> +        CLANG_LDFLAGS="-Wl,-flat_namespace -Wl,-undefined,suppress"
> +        dnl We are loaded into clang, so we don't need to link to very many things,
> +        dnl we just need to link to clangASTMatchers because it is not used by clang
> +        CLANG_LDFLAGS="$CLANG_LDFLAGS `$LLVMCONFIG --prefix`/lib/libclangASTMatchers.a"
> +        dnl We need to remove -L/path/to/clang/libfrom LDFLAGS to ensure that we

Nit: space between "lib" and "from".

@@ +58,5 @@
> +        CLANG_LDFLAGS="$CLANG_LDFLAGS `$LLVMCONFIG --prefix`/lib/libclangASTMatchers.a"
> +        dnl We need to remove -L/path/to/clang/libfrom LDFLAGS to ensure that we
> +        dnl don't accidentally link against the libc++ there which is a newer
> +        dnl version that what our build machines have installed.
> +        LLVM_LDFLAGS=`echo "$LLVM_LDFLAGS" | sed -E 's/-L[[^ ]]+\/clang\/lib//'`

Where did this value get inserted in the first place...I assuming from the above assignment to LLVM_LDFLAGS?  Maybe we just shouldn't run that in the first place, or only make the assignment on non-Darwin platforms?

Comment 46

[(no longer active)]

(In reply to Nathan Froyd [:froydnj] from comment #41)
> Comment on attachment 8832925 [details] [diff] [review]
> Part 4: Allow specifying a custom assembler and pass the right flags to it
> when cross-compiling
> 
> Review of attachment 8832925 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: build/build-clang/clang-static-analysis-win64.json
> @@ +10,5 @@
> >      "libcxx_repo": "https://llvm.org/svn/llvm-project/libcxx/trunk",
> >      "python_path": "c:/mozilla-build/python/python.exe",
> >      "cc": "cl.exe",
> >      "cxx": "cl.exe",
> > +    "ml": "ml64.exe",
> 
> Are we not updating the clang-static-analysis-win32.json file as well?

There's no need.  The "ml" default works for Win32.

Comment 47

[(no longer active)]

(In reply to Nathan Froyd [:froydnj] from comment #43)
> Comment on attachment 8832931 [details] [diff] [review]
> Part 10: Upgrade cctools used for building Firefox on OS X debug static
> analysis for support for LC_DATA_IN_CODE sections
> 
> Review of attachment 8832931 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: browser/config/mozconfigs/macosx64/debug-static-analysis
> @@ +3,5 @@
> >  MOZ_AUTOMATION_L10N_CHECK=0
> >  
> > +# The toolchain installed on our OSX 10.7 build machines is too old to support
> > +# MachO LC_DATA_IN_CODE load command, which newer LLVM generates, so we need to
> > +# use a newer toolchain that we build.
> 
> We don't need these assignments for the opt-static-analysis mozconfig?  Why
> not?

The opt builds run on Linux.  I was assuming you were planning to update that alongside the rest of the static analysis builders?  If not I can do that in a follow-up.

What's the bug number for the rest of the static analysis builds?  I was looking for it just now and couldn't find it.

Comment 48

[(no longer active)]

(In reply to Nathan Froyd [:froydnj] from comment #45)
> @@ +58,5 @@
> > +        CLANG_LDFLAGS="$CLANG_LDFLAGS `$LLVMCONFIG --prefix`/lib/libclangASTMatchers.a"
> > +        dnl We need to remove -L/path/to/clang/libfrom LDFLAGS to ensure that we
> > +        dnl don't accidentally link against the libc++ there which is a newer
> > +        dnl version that what our build machines have installed.
> > +        LLVM_LDFLAGS=`echo "$LLVM_LDFLAGS" | sed -E 's/-L[[^ ]]+\/clang\/lib//'`
> 
> Where did this value get inserted in the first place...I assuming from the
> above assignment to LLVM_LDFLAGS?

Yeah, this comes from llvm-config --ldflags.

> Maybe we just shouldn't run that in the
> first place, or only make the assignment on non-Darwin platforms?

I'd rather not do that, since there may be things in LDFLAGS that we want.  I think the right approach is using it all except for what we know is inappropriate.  On OS X, for example, this returns |-L/Users/cltbld/build/src/clang/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names|.

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a01a373da6a8
Part 1: Update the clang OSX build config file for cross-compile builds on the infrastructure; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/d44a0be29686
Part 2: Add cross compile jobs for clang 3.9 OSX builds; r=dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/a53ecc295eae
Part 3: Delete some unused patches; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/ffd70d960b9f
Part 4: Allow specifying a custom assembler and pass the right flags to it when cross-compiling; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/bbcae89fd7c2
Part 5: Add `-target x86_64-apple-darwin10' to the compiler-rt overridden CFLAGS; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/fc0b3a0ad1e5
Part 6: Reapply LLVM r277806 since our clang-plugin depends on it; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/a84b26a07369
Part 7: Reapply LLVM r285657 to work around bug 1291397 on clang 3.9; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/ccefeba51cf6
Part 8: Build cctools-port in the infrastructure; r=froydnj,dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/6631f3582920
Part 9: Upgrade cctools used for building clang on OS X for ld 264.3.102; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/f74c7a543525
Part 10: Upgrade cctools used for building Firefox on OS X debug static analysis for support for LC_DATA_IN_CODE sections; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/82c98214febe
Part 11: Specify the path to the cross-compiler toolchain libtool for OS X static analysis builds; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/a10f7732a754
Part 12: Remove -Lclang/lib from the LDFLAGS used to build the clang plugin; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/9a64d3102f6b
Part 13: Upgrade OSX static anlaysis builds to clang 3.9; r=froydnj

Comment 50

[Iris Hsiao [:ihsiao]]

Sorry had to back this out for OS X 10.7 debug static analysis build bustage, i.e., https://treeherder.mozilla.org/logviewer.html#?job_id=74132686&repo=mozilla-inbound&lineNumber=2414

https://hg.mozilla.org/integration/mozilla-inbound/rev/2a51c94218f2

Comment 51

[Nathan Froyd [:froydnj]]

(In reply to :Ehsan Akhgari from comment #47)
> The opt builds run on Linux.  I was assuming you were planning to update
> that alongside the rest of the static analysis builders?  If not I can do
> that in a follow-up.
> 
> What's the bug number for the rest of the static analysis builds?  I was
> looking for it just now and couldn't find it.

I don't think we have one.

Comment 52

[(no longer active)]

(In reply to Iris Hsiao [:ihsiao] from comment #50)
> Sorry had to back this out for OS X 10.7 debug static analysis build
> bustage, i.e.,
> https://treeherder.mozilla.org/logviewer.html#?job_id=74132686&repo=mozilla-
> inbound&lineNumber=2414
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/2a51c94218f2

Hmm, this makes no sense.  My only theory is that this is the issue.  During configure in the log we have:

17:52:51     INFO -  checking for has with ignoringParenImpCasts... (cached) no

Oops, clang 3.9 should return a yes response.  Indeed on try I got:

06:50:26     INFO -  checking for has with ignoringParenImpCasts... yes

So I bet this just needs a clobber.

Comment 53

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/07a36517afb8
Part 1: Update the clang OSX build config file for cross-compile builds on the infrastructure; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/7b0c267187e2
Part 2: Add cross compile jobs for clang 3.9 OSX builds; r=dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/45306965df08
Part 3: Delete some unused patches; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/119cde0c9f52
Part 4: Allow specifying a custom assembler and pass the right flags to it when cross-compiling; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/d04093c68e7c
Part 5: Add `-target x86_64-apple-darwin10' to the compiler-rt overridden CFLAGS; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/c31fbc8ee51a
Part 6: Reapply LLVM r277806 since our clang-plugin depends on it; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/8873211d237b
Part 7: Reapply LLVM r285657 to work around bug 1291397 on clang 3.9; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/41e7f431c83d
Part 8: Build cctools-port in the infrastructure; r=froydnj,dustin
https://hg.mozilla.org/integration/mozilla-inbound/rev/0593b423dc50
Part 9: Upgrade cctools used for building clang on OS X for ld 264.3.102; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/13413bf54f41
Part 10: Upgrade cctools used for building Firefox on OS X debug static analysis for support for LC_DATA_IN_CODE sections; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/4e58c4d85c94
Part 11: Specify the path to the cross-compiler toolchain libtool for OS X static analysis builds; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/eaeb4e28fa99
Part 12: Remove -Lclang/lib from the LDFLAGS used to build the clang plugin; r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/d8dd09d7df1d
Part 13: Upgrade OSX static anlaysis builds to clang 3.9; r=froydnj

Comment 54

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/07a36517afb8
https://hg.mozilla.org/mozilla-central/rev/7b0c267187e2
https://hg.mozilla.org/mozilla-central/rev/45306965df08
https://hg.mozilla.org/mozilla-central/rev/119cde0c9f52
https://hg.mozilla.org/mozilla-central/rev/d04093c68e7c
https://hg.mozilla.org/mozilla-central/rev/c31fbc8ee51a
https://hg.mozilla.org/mozilla-central/rev/8873211d237b
https://hg.mozilla.org/mozilla-central/rev/41e7f431c83d
https://hg.mozilla.org/mozilla-central/rev/0593b423dc50
https://hg.mozilla.org/mozilla-central/rev/13413bf54f41
https://hg.mozilla.org/mozilla-central/rev/4e58c4d85c94
https://hg.mozilla.org/mozilla-central/rev/eaeb4e28fa99
https://hg.mozilla.org/mozilla-central/rev/d8dd09d7df1d