727436 - Include compiler-rt with clang on Mac/Linux slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Christian Holler (:decoder)]

Right now, Clang 3.0 Trunk builds are already present on OSX and Linux slaves. In order to support address sanitizer in the future, we also need the compiler-rt to be installed there.

The ASan build instructions describe this step during building LLVM/Clang: https://developer.mozilla.org/en/Building_Firefox_with_Address_Sanitizer#LLVM.2FClang

Installing ASan separately should no longer be required and I hope that I can also get the Clang patch out of there and have that solved differently by upstream instead.

Comment 1

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Update clang and include compiler-rt

The attached patch updates clang, includes compiler rt and updates the configure options.
*) Updating clang to get more relaxed warning for default labels in a switch stament. Should make it easier to switch on OS X where we already use -Werror in some directories.
*) Include compiler-rt so that asan can use it.
*) Update the configure options since the support for using a non-default libstdc++ has been simplified.

Comment 2

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Forgot to mention. I was able to build firefox with it on OS X, stage2 of the 64 bit rpm is currently building and I have just started the 32 bit bootstrap.

Comment 3

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: update the clang we build on OS X

Comment 4

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: puppet manifest

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 597486 [details] [diff] [review]
puppet manifest

lgtm

Once you are ready, you can land rpm/dmg patches (not this one). Then I'll grab this bug, generate binaries and deploy using this patch.

Comment 6

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 597469 [details] [diff] [review]
update the clang we build on OS X

https://hg.mozilla.org/build/braindump/rev/461f5ebcc32a

Comment 7

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 597439 [details] [diff] [review]
Update clang and include compiler-rt

https://hg.mozilla.org/build/rpm-sources/rev/33186ef74ac7

Comment 8

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: pupet manifest that install only on 10.6

Comment 9

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I have so far been able to:
* bootstrap on linux 32 bits
* bootstrap on linux 64 bits
* build firefox on linux 32 bits
* build universal OS X
* build 64 bit debug builds on OS X

I have a 32 bit debug on OS X and a 64 bit build on linux running, but so far they look good.

I will do a try run once the packages are installed.

Comment 10

[Rail Aliiev [:rail]]

Assigning back to Rafael (there are some bugs in the current version, per IRC).

Comment 11

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Yes, it is llvm.org/pr12018 which only shows up on 32 bit debug builds :-(

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 597517 [details] [diff] [review]
pupet manifest that install only on 10.6

versions will be changed, removing r?

Comment 13

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: update clang to 150886 on linux

This revision has build firefox universal, 32 bit debug and 64 bit debug on OS X. I am currently building the 64 and 32 bit rpms.

Comment 14

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: update clang to 150886 on os x

Comment 15

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: puppet manifest for r150886

Comment 16

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Both rpm bulids and 4 firefox builds opt/debug X 32/64 bits finished correctly, so I think this rev is good to go.

Comment 17

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 599129 [details] [diff] [review]
update clang to 150886 on os x

https://hg.mozilla.org/build/braindump/rev/4b6c568c3582

Comment 18

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 599128 [details] [diff] [review]
update clang to 150886 on linux

https://hg.mozilla.org/build/rpm-sources/rev/2600c0025833

Comment 19

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: remove cpuid.h

Comment 20

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 599622 [details] [diff] [review]
remove cpuid.h

https://hg.mozilla.org/build/braindump/rev/cee4cdf16220

Comment 21

[Rail Aliiev [:rail]]

BTW, 150886 fails compiling on linux32 with the following error:

COMPILE:   clang_linux/full-x86_64/x86_64: /usr/src/redhat/SOURCES/llvm/projects/compiler-rt/lib/absvti2.c
Assembler messages:
Fatal error: No compiled in support for x86_64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[5]: *** [/usr/src/redhat/BUILD/clang-3.0-r150886.moz0/stage1/tools/clang/runtime/compiler-rt/clang_linux/full-x86_64/x86_64/SubDir.lib/absvdi2.o] Error 1
make[5]: *** Waiting for unfinished jobs....
Assembler messages:
Fatal error: No compiled in support for x86_64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[5]: *** [/usr/src/redhat/BUILD/clang-3.0-r150886.moz0/stage1/tools/clang/runtime/compiler-rt/clang_linux/full-x86_64/x86_64/SubDir.lib/absvsi2.o] Error 1
Assembler messages:
Fatal error: No compiled in support for x86_64
clang: error: assembler command failed with exit code 1 (use -v to see invocation)
make[5]: *** [/usr/src/redhat/BUILD/clang-3.0-r150886.moz0/stage1/tools/clang/runtime/compiler-rt/clang_linux/full-x86_64/x86_64/SubDir.lib/absvti2.o] Error 1
make[5]: Leaving directory `/usr/src/redhat/SOURCES/compiler-rt'
make[4]: *** [BuildRuntimeLibraries] Error 2
rm /usr/src/redhat/BUILD/clang-3.0-r150886.moz0/stage1/Release+Asserts/lib/clang/3.1/lib/linux/.dir
make[4]: Leaving directory `/usr/src/redhat/BUILD/clang-3.0-r150886.moz0/stage1/tools/clang/runtime/compiler-rt'
make[3]: *** [compiler-rt/.makeall] Error 2
make[3]: Leaving directory `/usr/src/redhat/BUILD/clang-3.0-r150886.moz0/stage1/tools/clang/runtime'
make[2]: *** [all] Error 1
make[2]: Leaving directory `/usr/src/redhat/BUILD/clang-3.0-r150886.moz0/stage1/tools/clang'
make[1]: *** [clang/.makeall] Error 2

Comment 22

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to Rail Aliiev [:rail] from comment #21)
> BTW, 150886 fails compiling on linux32 with the following error:

> /usr/src/redhat/SOURCES/llvm/projects/compiler-rt/lib/absvti2.c
> Assembler messages:
> Fatal error: No compiled in support for x86_64
> clang: error: assembler command failed with exit code 1 (use -v to see
> invocation)

Looks like we are missing the 64 bit assembler on the bots and for some reason clang is not using the integrated one. I will try to reproduce the failure, but it worked for me the previous time. Can you try running
make VERBOSE=1
to see what is the command line that is failing?

Comment 23

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I am getting

/usr/src/redhat/BUILD/clang-3.0-r151245.moz0/stage1/Release+Asserts/bin/clang -no-integrated-as -Wall -Werror -O3 -fomit-frame-pointer -m64 --sysroot=/usr/src/redhat/SOURCES/llvm/projects/compiler-rt/SDKs/linux -c -fPIC -o /usr/src/redhat/BUILD/clang-3.0-r151245.moz0/stage1/tools/clang/runtime/compiler-rt/clang_linux/full-x86_64/x86_64/SubDir.lib/absvti2.o /usr/src/redhat/SOURCES/llvm/projects/compiler-rt/lib/absvti2.c

so we are using the system assembler, but it looks like the one I have supports 64 bits. It is:

/usr/bin/as --version
GNU assembler 2.17.50.0.6-14.el5 20061020

rpm -qf /usr/bin/as
binutils-2.17.50.0.6-14.el5

Which version do you see in that bot?

Comment 24

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Update clang to 151367 on linux

I am going to do the normal bootstrap and firefox building tests. Can you try building this on a 32 bit bot? They have an assembler that is older than the one I got in my centos 5 vm :-(

Comment 25

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Update clang to 151367 on os x

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 600406 [details] [diff] [review]
Update clang to 151367 on linux

http://hg.mozilla.org/build/rpm-sources/rev/a967e43c2317

Comment 27

[Rail Aliiev [:rail]]

Comment on attachment 600407 [details] [diff] [review]
Update clang to 151367 on os x

http://hg.mozilla.org/build/braindump/rev/79db3c80df4d

Comment 28

[Rail Aliiev [:rail]]

Attachment: puppet-manifests

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 600610 [details] [diff] [review]
puppet-manifests

Rafael, if you're happy with the current revision, I can deploy it early next week.

BTW, linux32 build worked fine with the same version of assembler you use.

Comment 30

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to Rail Aliiev [:rail] from comment #29)
> Comment on attachment 600610 [details] [diff] [review]
> puppet-manifests
> 
> Rafael, if you're happy with the current revision, I can deploy it early
> next week.

Yes, this version passed basic testing for me.

> BTW, linux32 build worked fine with the same version of assembler you use.

It should not be using the assembler anymore, so it should work with the old one too.

Comment 31

[Rail Aliiev [:rail]]

Comment on attachment 600610 [details] [diff] [review]
puppet-manifests

http://hg.mozilla.org/build/puppet-manifests/rev/063a1aba7648

Comment 32

[Rail Aliiev [:rail]]

The package should be available on all build machines by tomorrow.