1253299 - TC Linux 64 ASAN debug and opt builds Tier 1

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Selena Deckelmann :selenamarie :selena]

This is a planning bug for tracking work related to getting Linux 64 ASAN debug and opt builds as Tier-1.

Comment 1

[Chris AtLee [:catlee]]

Attachment: MozReview Request: Bug 1253299 - Add linux64 debug/opt asan

WIP

Review commit: https://reviewboard.mozilla.org/r/38719/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/38719/

Comment 2

[Chris AtLee [:catlee]]

https://reviewboard.mozilla.org/r/38719/#review35397

This is just a WIP for now.

Right now this fails to build on both opt and debug because clang can't seem to link things properly:

https://tools.taskcluster.net/task-inspector/#ecmw6igNTIC-J-zz36y67A/0

Comment 3

[Geoff Brown [:gbrown]]

Attachment: catlee's patch updated for bitrot

catlee's patch was out of date because of _clobber changes. This just adjusts for that. I get the same clang failure:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=76405d61a70e

19:34:18     INFO -  DEBUG: configure:3123: checking whether the C compiler (/home/worker/workspace/build/src/clang/bin/clang -fgnu89-inline -fsanitize=address -Dxmalloc=myxmalloc -fPIC -L/home/worker/workspace/build/src/gtk3/usr/local/lib -fsanitize=address) works
19:34:18     INFO -  DEBUG: configure:3139: /home/worker/workspace/build/src/clang/bin/clang -fgnu89-inline -o conftest -fsanitize=address -Dxmalloc=myxmalloc -fPIC  -L/home/worker/workspace/build/src/gtk3/usr/local/lib -fsanitize=address conftest.c  1>&5
19:34:18     INFO -  DEBUG: configure:3136:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
19:34:18     INFO -  DEBUG: main(){return(0);}
19:34:18     INFO -  DEBUG: ^~~~
19:34:18     INFO -  DEBUG: 1 warning generated.
19:34:18     INFO -  DEBUG: /usr/bin/ld: crtbegin.o: No such file: No such file or directory
19:34:18     INFO -  DEBUG: clang: error: linker command failed with exit code 1 (use -v to see invocation)
19:34:18     INFO -  DEBUG: configure: failed program was:
19:34:18     INFO -  DEBUG:
19:34:18     INFO -  DEBUG: #line 3134 "configure"
19:34:18     INFO -  DEBUG: #include "confdefs.h"
19:34:18     INFO -  DEBUG:
19:34:18     INFO -  DEBUG: main(){return(0);}
19:34:18     INFO -  DEBUG: configure: error: installation or configuration problem: C compiler cannot create executables.
19:34:18     INFO -  ERROR: old-configure failed
19:34:18     INFO -  *** Fix above errors and then restart with\
19:34:18     INFO -                 "/usr/bin/gmake -f client.mk build"
19:34:18     INFO -  gmake[2]: *** [configure] Error 1


/home/worker/workspace/build/src/clang/bin/clang appears to come from tooltool, same as on buildbot, and clang command line options appear to be the same as buildbot.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

odd, maybe "/usr/bin/ld: crtbegin.o: No such file: No such file or directory", which would imply that we didn't get output from the compiler and ld is obviously failing.

maybe this is a permissions issue?  maybe a disk space issue?  I suspect downloading the image.tar and running it locally would yield a quick answer!  

Thanks for working on this bug.

Comment 5

[Geoff Brown [:gbrown]]

crtbegin.o is in the image, at:

  /usr/lib/gcc/x86_64-redhat-linux/4.4.4/32/crtbegin.o
  /usr/lib/gcc/x86_64-redhat-linux/4.4.4/crtbegin.o

but clang doesn't seem to know that gcc is installed there:

23:24:51     INFO - Copy/paste: /home/worker/workspace/build/src/clang/bin/clang --print-search-dirs
23:24:51     INFO -  programs: =/home/worker/workspace/build/src/clang/bin:/..//bin
23:24:51     INFO -  libraries: =/home/worker/workspace/build/src/clang/bin/../lib/clang/3.5:/lib/../lib64:/usr/lib/../lib64:/home/worker/workspace/build/src/clang/bin/../lib:/lib:/usr/lib
23:24:51     INFO - Return code: 0

23:24:51     INFO - Copy/paste: /home/worker/workspace/build/src/clang/bin/clang --print-libgcc-file-name
23:24:51     INFO -  libgcc.a
23:24:51     INFO - Return code: 0

23:24:51     INFO - Copy/paste: /home/worker/workspace/build/src/clang/bin/clang -v
23:24:51     INFO -  clang version 3.5 (trunk 200213)
23:24:51     INFO -  Target: x86_64-unknown-linux-gnu
23:24:51     INFO -  Thread model: posix
23:24:51     INFO -  Selected GCC installation:
23:24:51     INFO - Return code: 0


Happily, I notice that clang is already used in a tc job -- static analysis. One difference is it uses a different clang manifest:

'tooltool_manifest_src': 'browser/config/tooltool-manifests/linux64/clang.manifest.centos6'

instead of:

'tooltool_manifest_src': 'browser/config/tooltool-manifests/linux64/asan.manifest'

Comment 6

[Geoff Brown [:gbrown]]

Simply changing to the clang.manifest.centos6 allows clang to find gcc, and most of the build completes.

16:45:22     INFO - Copy/paste: /home/worker/workspace/build/src/clang/bin/clang --print-search-dirs
16:45:22     INFO -  programs: =/home/worker/workspace/build/src/clang/bin:/home/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.7.3/../../../../x86_64-unknown-linux-gnu/bin
16:45:22     INFO -  libraries: =/home/worker/workspace/build/src/clang/bin/../lib/clang/3.8.0:/home/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.7.3:/lib/../lib64:/usr/lib/../lib64:/home/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.7.3/../../..:/home/worker/workspace/build/src/clang/bin/../lib:/lib:/usr/lib
16:45:22     INFO - Return code: 0

16:45:22     INFO - Copy/paste: /home/worker/workspace/build/src/clang/bin/clang --print-libgcc-file-name
16:45:22     INFO -  /home/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.7.3/libgcc.a
16:45:22     INFO - Return code: 0

16:45:22     INFO - Copy/paste: /home/worker/workspace/build/src/clang/bin/clang -v
16:45:22     INFO -  clang version 3.8.0 (trunk) (llvm/trunk 247539)
16:45:22     INFO -  Target: x86_64-unknown-linux-gnu
16:45:22     INFO -  Thread model: posix
16:45:22     INFO -  InstalledDir: /home/worker/workspace/build/src/clang/bin
16:45:22     INFO -  Found candidate GCC installation: /home/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.7.3
16:45:22     INFO -  Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.4.4
16:45:22     INFO -  Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/4.4.7
16:45:22     INFO -  Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.4
16:45:22     INFO -  Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.7
16:45:22     INFO -  Selected GCC installation: /home/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.7.3
16:45:22     INFO -  Candidate multilib: .;@m64
16:45:22     INFO -  Candidate multilib: 32;@m32
16:45:22     INFO -  Selected multilib: .;@m64
16:45:22     INFO - Return code: 0

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1c1d60b0cce2

17:12:26     INFO -  /home/worker/workspace/build/src/clang/bin/clang -fgnu89-inline -o /home/worker/workspace/build/src/obj-firefox/security/nss/cmd/shlibsign/shlibsign -O2 -gdwarf-2 -fPIC -DLINUX2_1 -m64 -pipe -ffunction-sections -fdata-sections -DLINUX -Dlinux -DHAVE_STRERROR -Wall -DNSS_NO_GCC48 -DXP_UNIX -DSHLIB_SUFFIX=\"so\" -DSHLIB_PREFIX=\"lib\" -UDEBUG -DNDEBUG -D_REENTRANT -DUSE_UTIL_DIRECTLY -DNO_NSPR_10_SUPPORT -DSSL_DISABLE_DEPRECATED_CIPHER_SUITE_NAMES -I/home/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/home/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/home/worker/workspace/build/src/obj-firefox/dist/include/nss -I/home/worker/workspace/build/src/obj-firefox/dist/private/nss -Qunused-arguments -Qunused-arguments -fsanitize=address -Dxmalloc=myxmalloc -fPIC -std=gnu99 -fgnu89-inline -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe -g -O2 -gline-tables-only -fno-omit-frame-pointer /home/worker/workspace/build/src/obj-firefox/security/nss/cmd/shlibsign/shlibsign.o -m64  -L/home/worker/workspace/build/src/obj-firefox/dist/bin -lplc4 -lplds4 -lnspr4  -lpthread  -ldl -lc
17:12:26     INFO -  /usr/bin/ld: /home/worker/workspace/build/src/obj-firefox/security/nss/cmd/shlibsign/shlibsign: undefined reference to symbol '__cxa_demangle@@CXXABI_1.3'
17:12:26     INFO -  /usr/bin/ld: note: '__cxa_demangle@@CXXABI_1.3' is defined in DSO /usr/lib64/libstdc++.so.6 so try adding it to the linker command line
17:12:26     INFO -  /usr/lib64/libstdc++.so.6: could not read symbols: Invalid operation
17:12:26     INFO -  clang-3.8: error: linker command failed with exit code 1 (use -v to see invocation)
17:12:26     INFO -  gmake[6]: *** [/home/worker/workspace/build/src/obj-firefox/security/nss/cmd/shlibsign/shlibsign] Error 1

Comment 7

[Geoff Brown [:gbrown]]

I don't understand the cause of the link error in Comment 6:

/home/worker/workspace/build/src/obj-firefox/security/nss/cmd/shlibsign/shlibsign: undefined reference to symbol '__cxa_demangle@@CXXABI_1.3'

The link error goes away if I suppress asan (remove -fsanitize=address and --enable-address-sanitizer):

https://treeherder.mozilla.org/#/jobs?repo=try&revision=358066c393bd

but that's not a helpful work-around!

:ehsan -- I'm trying to use clang from browser/config/tooltool-manifests/linux64/clang.manifest.centos6; is that reasonable? Do you have insight into the link error / how to avoid it?

Comment 8

[(no longer active)]

(In reply to Geoff Brown [:gbrown] from comment #7)
> I don't understand the cause of the link error in Comment 6:
> 
> /home/worker/workspace/build/src/obj-firefox/security/nss/cmd/shlibsign/
> shlibsign: undefined reference to symbol '__cxa_demangle@@CXXABI_1.3'
> 
> The link error goes away if I suppress asan (remove -fsanitize=address and
> --enable-address-sanitizer):
> 
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=358066c393bd
> 
> but that's not a helpful work-around!
> 
> :ehsan -- I'm trying to use clang from
> browser/config/tooltool-manifests/linux64/clang.manifest.centos6; is that
> reasonable? Do you have insight into the link error / how to avoid it?

Can you please retry on top of bug 1262735?  What version of clang are you using?

To answer your question: I think this error means that you don't have the right libc++abi version installed.

Comment 9

[Geoff Brown [:gbrown]]

(In reply to :Ehsan Akhgari (busy, don't ask for review please) from comment #8)
> Can you please retry on top of bug 1262735?  What version of clang are you using?

Good idea. Thanks Ehsan.

But I still get the same failure on top of bug 1262735 / bug 1262781, using browser/config/tooltool-manifests/linux64/clang.manifest.centos6 to pull clang from tooltool with version tag "clang 3.8.0, libgcc 4.8.5".

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f936f328e9aa

21:18:56     INFO -  /usr/bin/ld: /home/worker/workspace/build/src/obj-firefox/security/nss/cmd/shlibsign/shlibsign: undefined reference to symbol '__cxa_demangle@@CXXABI_1.3'

Comment 10

[Geoff Brown [:gbrown]]

Attachment: work in progress - Add linux64 debug/opt asan

Comment 11

[Kim Moir [:kmoir] ET]

Ran a try job this morning and it failed because the decision graph couldn't be generated, 
investigating
https://treeherder.mozilla.org/#/jobs?repo=try&revision=024ce62ee9848a73cd9c1b29fc9f6fdf80e3cbcc

Comment 12

[Kim Moir [:kmoir] ET]

so this submission is better
https://treeherder.mozilla.org/#/jobs?repo=try&revision=07e5007647c4

I omitted committing new files in the patch in the earlier run

Comment 13

[Kim Moir [:kmoir] ET]

Same error on that try run as last.  Looking at the manifests, the one we used for the try run is 
[
{
"version": "clang 3.8.0, libgcc 4.8.5",
"size": 118876936,
"digest": "f021d7b23cbbcc4086514b4bf20c86c9f94dbe5b4e0f1ef3aaf2bea337430f7c0d0965c55fe8be0e944a46c3b1555b9245c7af8b8b141eac4b47deea977b4852",
"algorithm": "sha512",
"filename": "clang.tar.xz",
"unpack": true
},
{
"size": 12072532,
"digest": "3915f8ec396c56a8a92e6f9695b70f09ce9d1582359d1258e37e3fd43a143bc974410e4cfc27f500e095f34a8956206e0ebf799b7287f0f38def0d5e34ed71c9",
"algorithm": "sha512",
"filename": "gtk3.tar.xz",
"setup": "setup.sh",
"unpack": true
}
]

old one for bb builds in tree is

[
{
"version": "clang 3.5/r200213",
"size": 71282740,
"digest": "ee9edb1ef3afd9ab29e39565145545ad57e8d8d2538be4d822d7dbd64038f4529b0b287cecf48bf83def52a26ac2c6faa331686c3ad5e8b4ba4c22686ee0808f",
"algorithm": "sha512",
"filename": "clang.tar.bz2",
"unpack": true
},
{
"size": 12072532,
"digest": "3915f8ec396c56a8a92e6f9695b70f09ce9d1582359d1258e37e3fd43a143bc974410e4cfc27f500e095f34a8956206e0ebf799b7287f0f38def0d5e34ed71c9",
"algorithm": "sha512",
"filename": "gtk3.tar.xz",
"setup": "setup.sh",
"unpack": true
}
]

I'm comaparing the two tooltool clang binaries and seeing if there are any glaring differences, if not, I'll ask for help from a build person

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Note that you can also look up information about the tooltool uploads (comment, who uploaded, datestamp) in the relengapi UI

Comment 16

[Christian Holler (:decoder)]

Looking at the patch attached to this bug, it seems to me that you are planning to remove the manifest for ASan and instead use the regular Clang manifest that we use for our release builds, is that correct?

If that is the case, please keep the separate manifest. We introduced it for a particular reason and that is that upgrading the ASan Clang version was nearly impossible when it was still tied to the regular Clang version. If someone commits to keeping Clang up-to-date in general, that's a different story though.

I also filed bug 1267650 to upgrade the ASan Clang version to trunk (what will be 3.9). I made a local build and it works fine with some fix I already landed. It would be nice if we could get that version for ASan on TaskCluster, it would greatly help the fuzzing team. If I can help solving problems, let me know.

Comment 17

[Kim Moir [:kmoir] ET]

I'm trying to build a new version of clang as described in bug 1267650 comment 4

https://treeherder.mozilla.org/#/jobs?repo=try&revision=972837dd076b

Once this is ready I will try another asan job with it on try (and keep the separate manifest as described in comment 16)

Comment 18

[Mike Hommey [:glandium]]

(In reply to Kim Moir [:kmoir] from comment #17)
> I'm trying to build a new version of clang as described in bug 1267650
> comment 4
> 
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=972837dd076b
> 
> Once this is ready I will try another asan job with it on try (and keep the
> separate manifest as described in comment 16)

Since you didn't change anything in the tree, all you'll get there is the same clang as the one we already use (3.8).

Comment 19

[Kim Moir [:kmoir] ET]

So looking at a bug where we upgraded clang to 3.8.0 (bug 1262781) - we pulled from  (https://llvm.org/svn/llvm-project/cfe/tags/RELEASE_380/).  However, 3.9 is not released yet.  So should just try trunk and hope for the best or is there a specific revision that we could use similar to the 3.8 bug?

https://reviewboard.mozilla.org/r/45685/diff/1#8

Comment 20

[Mike Hommey [:glandium]]

If you really want a 3.9 development snapshot, pull from trunk with a specific revision and document the revision in the tooltool manifest (check how it was before bug 1262781).

That said, it would be better if we stopped using non-release versions of the toolchains. What's wrong with 3.8?

Comment 21

[Christian Holler (:decoder)]

(In reply to Mike Hommey [:glandium] from comment #20)
> If you really want a 3.9 development snapshot, pull from trunk with a
> specific revision and document the revision in the tooltool manifest (check
> how it was before bug 1262781).

But I still need to provide the necessary build tarball, right? I know how the tooltool manifests work but when I did the last Clang update on our infra, we still had to bug releng to upload a new tarball for us. Has that changed?

> 
> That said, it would be better if we stopped using non-release versions of
> the toolchains. What's wrong with 3.8?

ASan constantly gets improved and we want to have the newest ASan support available. Since 3.8, numerous fixes (mainly affecting Windows) have been made to compiler-rt. For Linux, 3.8 might actually suffice right now, but our experience is that we have to make fixes (to Clang) on such major upgrades and these fixes will end up on trunk only. We can try it with 3.8 first and then upgrade to trunk or 3.9 (if released), when we need it on Windows.

Comment 22

[Mike Hommey [:glandium]]

(In reply to Christian Holler (:decoder) from comment #21)
> (In reply to Mike Hommey [:glandium] from comment #20)
> > If you really want a 3.9 development snapshot, pull from trunk with a
> > specific revision and document the revision in the tooltool manifest (check
> > how it was before bug 1262781).
> 
> But I still need to provide the necessary build tarball, right? I know how
> the tooltool manifests work but when I did the last Clang update on our
> infra, we still had to bug releng to upload a new tarball for us. Has that
> changed?

You can ask for permission to upload to tooltool if necessary. And since bug 1262729, you can build linux toolchains by pushing to try.

> > That said, it would be better if we stopped using non-release versions of
> > the toolchains. What's wrong with 3.8?
> 
> ASan constantly gets improved and we want to have the newest ASan support
> available. Since 3.8, numerous fixes (mainly affecting Windows) have been
> made to compiler-rt. For Linux, 3.8 might actually suffice right now, but
> our experience is that we have to make fixes (to Clang) on such major
> upgrades and these fixes will end up on trunk only. We can try it with 3.8
> first and then upgrade to trunk or 3.9 (if released), when we need it on
> Windows.

Windows is using a snapshot of 3.9 already.

Comment 23

[Kim Moir [:kmoir] ET]

I built a new clang with the same revision that windows is using
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7867941909e8161f72d0969379c7fea3411fcdd0

Now trying a new asan build but it is failing
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a387b31fff67fca8fee37f3561d8404368a64c69

seems like clang is not put in the correct place, investigating 
https://public-artifacts.taskcluster.net/Hp3yI-UMSweAOhGxq7Jw8w/0/public/logs/live_backing.log

Comment 24

[Kim Moir [:kmoir] ET]

okay, now the issues with finding clang are fixed but the build itself still fails with compile errors
https://treeherder.mozilla.org/#/jobs?repo=try&revision=15debfd306fd1045c668cbcda821fd343873a789

(ignore the fact that the try run built clang that is just because of my patch queue)

taskcluster logs
https://public-artifacts.taskcluster.net/VITKbCVbT5O_ZmRPPO5Caw/0/public/logs/live_backing.log

This is clang revision 262971 (toward 3.9)

Not sure if/what patches are needed to make it work

Comment 25

[Kim Moir [:kmoir] ET]

:decoder do you have any insight on the source of the compile errors for this build? Not sure if additional patches are required for 3.9.  The existing ones for 3.8 in tree didn't apply anymore.

Comment 26

[Mike Hommey [:glandium]]

Why the f*ck does clang want to use a C++ symbol in a C program?

Comment 27

[Geoff Brown [:gbrown]]

That's a good question. :)

I'm not sure if this is a clue or a red herring, but maybe worth repeating at this point: When I was trying to get this working and getting the same error (comment 7), I found the error went away if I suppressed asan (removed -fsanitize=address and --enable-address-sanitizer).

Comment 28

[Christian Holler (:decoder)]

Well, AddressSanitizer itself (and its runtime) is written in C++, so that might explain where the C++ symbol came from, right?

We had issues before with our codebase due to C++ compile errors, but until now, --enable-stdcxx-compat resolved those.

I also vaguely remember that we sometimes had to add C++ functions to a certain compat list (?) in our tree?

Comment 29

[Kim Moir [:kmoir] ET]

Does anyone know where this compat list in the tree is for the C++ functions?

Comment 30

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/mozilla-central/annotate/c3f5e6079284/build/unix/stdc++compat/stdc%2B%2Bcompat.cpp ???

Comment 31

[Kim Moir [:kmoir] ET]

:decoder - Could you identify what needs to be updated in the compat file list mentioned in comment #30?  I don't have any expertise in this area.  I'm just trying to port this build to taskcluster.

Comment 32

[Christian Holler (:decoder)]

Forwarding this to :glandium since he should know best (and maintains this file, as far as I know).

Comment 33

[Mike Hommey [:glandium]]

Here is the catch: stdcxx-compat is only linked to C++ binaries. So even if adding something there was the right thing to do (which I doubt it would be anyways), it wouldn't change anything for those errors.

Now, what seems to be happening is:
- shlibsign is linked from clang, because it's a pure C program.
- As such, clang calls ld with the standard set of libraries for C programs (-lc. -lm, etc.) and *not* -lstdc++, which is for C++ programs
- because there is also -fsanitize=address, it also adds the asan library, which apparently wants symbols from -lstdc++
- ld sees that there's a symbol from libstdc++ is missing, and considers it
- it then later rejects linking libstdc++ because it was not on the command line

One way out of this is to manually add -lstdc++ to the LDFLAGS, but it seems ridiculous to me that clang is not doing it itself if -fsanitize=address requires it (clang bug?). Also, why is it not happening on the existing asan builds? Is the requirement for __cxa_demangle new?

Comment 34

[Mike Hommey [:glandium]]

per irc:

19:54 <glandium> decoder: so, there is no problem with clang, it more or less does the right thing
19:55 <glandium> decoder: that is, whatever uses __cxa_demangle is using it with a weak reference
19:56 <glandium> decoder: what happens is that because of the combination of bug 743988 and bug 1230117, we now link nspr with libstdc++
19:57 <glandium> decoder: when ld links shlibsign, it sees that it's pulling libstdc++ from nspr, sees that __cxa_demangle is there too, so hey, why not use it
19:57 <glandium> decoder: and then, it says, oh, no, in fact, go away, I don't want symbol references to things I'm not depending on directly
19:59 <glandium> decoder: it's likely related to https://fedoraproject.org/wiki/Features/ChangeInImplicitDSOLinking
20:00 <glandium> decoder: and why it happens on automation and not on your build is likely that it's not the same on ubuntu
20:00 <glandium> or whatever distro you use
20:01 <decoder> glandium: im on ubuntu 16.04 (and 14.04 is the same)
20:03 <glandium> decoder: another possible explanation for the discrepancy is that, since it's a weak symbol, ld shouldn't complain anyways, and maybe that part was 
                 fixed in later versions
20:04 <glandium> decoder: so there are two things that we can do to work around, each of which should work independently, but both of which should actually happen for 
                 various reasons:
20:04 <glandium> decoder: - link nspr with $CC instead of $CXX (kind of a backout of bug 743988)
20:05 <glandium> decoder: - include, along clang, the same binutils we have along gcc
20:05 <decoder> the latter sounds a better to me in some ways
20:05 <decoder> who knows what other bugs are in those old binutils
20:05 <glandium> decoder: both should be done
20:05 <decoder> and better performance
20:07 <glandium> decoder: the easiest for you is probably the latter
20:07 <glandium> I guess changing build_clang.py to copy a few files from the gcc package, like it does for libgcc and headers, would be enough
20:08 <glandium> or, we could create a separate package for binutils
20:08 <glandium> actually, I think I prefer the latter

I will file bugs for both items.

Comment 35

[Kim Moir [:kmoir] ET]

Attachment: bug1272629new-2.patch

new patches with fixes from bug 1272629. I used separate tooltool manifests for the tc and buildbot jobs since they use different versions of clang etc.  

The builds run green, now I will run with tests 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e262205e76cecc66330b07a3ad2764bc2ab1befb

Comment 36

[Kim Moir [:kmoir] ET]

The try run was green for buildbot tests except for tests that aren't enabled on asan yet anyways 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7b00f51c523f6aae82d08547775e792087f0f421

bug 1253301 tracks getting ASAN tests working on TC is in progress so I didn't add hooks to get them to run

Comment 37

[Kim Moir [:kmoir] ET]

Comment on attachment 8757285 [details] [diff] [review]
bug1272629new-2.patch

gbrown: I was going to add a second tool manifest to get the asan builds into production on tc and then deprecate the buildbot one once they are fully deployed on all trees - what are your thoughts?

Comment 38

[Joel Maher ( :jmaher ) (UTC -8)]

gbrown is on pto for 2 weeks- possibly there is another person to feedback?

Comment 39

[Kim Moir [:kmoir] ET]

thanks joel, will ask Mihai for feedback since he has been working on tc migrations lately

Comment 40

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8757285 [details] [diff] [review]
bug1272629new-2.patch

Review of attachment 8757285 [details] [diff] [review]:
-----------------------------------------------------------------

Haven't played yet with manifests so not sure how much my feedback helps there. However, testing/taskcluster/... stuff looks good.

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fd184690f95c
TC Linux 64 ASAN debug and opt builds Tier 1 r=mtabara

Comment 42

[Kim Moir [:kmoir] ET]

Comment on attachment 8757285 [details] [diff] [review]
bug1272629new-2.patch

http://hg.mozilla.org/integration/mozilla-inbound/rev/fd184690f95c

Comment 43

[Kim Moir [:kmoir] ET]

both tc and buildbot asan builds are green on inbound
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=fd184690f95c861e5e492cae580756afdb554f09&selectedJob=29217331

Comment 44

[Joel Maher ( :jmaher ) (UTC -8)]

excellent!

Comment 45

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/fd184690f95c

Comment 46

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Kim, it looks like you accidentally landed an orig file together with your remaining patch:

https://hg.mozilla.org/mozilla-central/diff/fd184690f95c/testing/taskcluster/tasks/branches/base_job_flags.yml.orig

Comment 47

[Kim Moir [:kmoir] ET]

I'll land the fix for this as soon as trees reopen.

Comment 48

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4724bebdf93a
TC Linux 64 ASAN debug and opt builds Tier 1 r=mtabara DONTBUILD

Comment 49

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4724bebdf93a