957865 - Upgrade ASan Clang in Q1

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Christian Holler (:decoder)]

We are considering to upgrade Clang for ASan to a more recent version in Q1. One of the main reasons is support for TSan (ThreadSanitizer), but also certain fixes/optimizations made for ASan. We don't have a target version yet, but I will try to make myself familiar first with the necessary build steps so we can provide the binaries to deploy.

Comment 1

[Rail Aliiev [:rail]]

See also https://wiki.mozilla.org/ReleaseEngineering/How_To/Clang_update#Compilation. It describes how to generate the binaries using your own CentOS or using vagrant images.

Comment 2

[Rail Aliiev [:rail]]

Per :decoder's request I uploaded the following file to tooltool:
 http://users.own-hero.net/~decoder/b5b2479ccad6687cdf4485e8bf3cdb289916e5555dbc8dbcd95d952e349f6eed45a3f6f3505e40dfc8abc62579ebb50d73dcdeebbd6519c4c3f3074176a4b3ed

size: 70800172

sha512: b5b2479ccad6687cdf4485e8bf3cdb289916e5555dbc8dbcd95d952e349f6eed45a3f6f3505e40dfc8abc62579ebb50d73dcdeebbd6519c4c3f3074176a4b3ed

Comment 3

[Christian Holler (:decoder)]

Attachment: ASan Clang manifest for r199443. (WIP, for testing)

This is the manifest to be used with Rail's upload.

Comment 4

[Christian Holler (:decoder)]

We have a new Clang version available now for testing with ASan (see manifest above), but you might as well want to use the opportunity to test this without ASan, in case you want to upgrade Clang in the future.

Comment 5

[Christian Holler (:decoder)]

Green ASan try runs \o/

Opt build + tests: https://tbpl.mozilla.org/?tree=Try&rev=f0dc91c240cc
Debug build: https://tbpl.mozilla.org/?tree=Try&rev=abf760f17f08

Comment 6

[Christian Holler (:decoder)]

So far the build does great (no ASan regression and TSan is working as it seems).

However, I'll likely do another update/build to include this fix that landed today:

http://code.google.com/p/thread-sanitizer/issues/detail?id=46

With TSan, I noticed a few times that Firefox was hanging/had a stale process, and this could be associated.

Comment 7

[Christian Holler (:decoder)]

Attachment: build-clang-trunk.patch

Build script changes to build Clang Trunk. Changes made:

* Changed build option -j8 to -j4 because the Vagrant box will OOM during the build when using -j8 with the newest Clang. -j4 works cleanly.

* Use the newer moz-gcc/g++ when building Clang, as Clang now requires at least GCC 4.7. This also requires to set the LD_LIBRARY_PATH appropriately when building with that custom gcc.

* The no-sse-on-linux.patch either applies to trunk or 3.3 so I made a copy of it for the trunk version. The second patch applies cleanly to all versions.

Comment 8

[Christian Holler (:decoder)]

Attachment: ASan Clang manifest for r200213. (WIP, for testing)

Comment 9

[Christian Holler (:decoder)]

Comment on attachment 8365960 [details] [diff] [review]
ASan Clang manifest for r200213. (WIP, for testing)

Rail, I've made some more tests with r200213 and I'm quite confident that we can stick to this revision for ASan/TSan. We're only doing 64 bit testing anyway, so I guess we don't need the 32 bit asan.manifest. However, Mac OSX might be interesting in the future and I don't know how to create a package for that. Can you maybe create the Mac package so we can land the manifest with changes to linux64+mac asan.manifest?

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8365960 [details] [diff] [review]
ASan Clang manifest for r200213. (WIP, for testing)

I started a mac build, but I had to add --disable-compiler-version-checks to the ./configure call -- looks like clang from Xcode is too old. Not sure if this is a problem because we build in multiple stages...

Comment 11

[Rail Aliiev [:rail]]

Hmmmm, it didn't work and failed

  MKDIR:     /builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_macho_embedded/hard_pic/x86_64
  MKDIR:     /builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_macho_embedded/hard_pic
  FINAL-ARCHIVE: clang_darwin/eprintf: /builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_darwin/eprintf/libcompiler_rt.a
lipo: no input files specified
lipo: Usage: lipo [input_file] ... [-arch <arch_type> input_file] ... [-info] [-detailed_info] [-output output_file] [-create] [-arch_blank <arch_type>] [-thin <arch_type>] [-remove <arch_type>] ... [-extract <arch_type>] ... [-extract_family <arch_type>] ... [-verify_arch <arch_type> ...] [-replace <arch_type> <file_name>] ...
make[5]: *** [/builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_darwin/eprintf/libcompiler_rt.a] Error 1
make[5]: *** Waiting for unfinished jobs....
  FINAL-ARCHIVE: clang_darwin/10.4: /builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_darwin/10.4/libcompiler_rt.a
  FINAL-ARCHIVE: clang_darwin/ios: /builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_darwin/ios/libcompiler_rt.a
lipo: no input files specified
lipo: Usage: lipo [input_file] ... [-arch <arch_type> input_file] ... [-info] [-detailed_info] [-output output_file] [-create] [-arch_blank <arch_type>] [-thin <arch_type>] [-remove <arch_type>] ... [-extract <arch_type>] ... [-extract_family <arch_type>] ... [-verify_arch <arch_type> ...] [-replace <arch_type> <file_name>] ...
make[5]: *** [/builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_darwin/10.4/libcompiler_rt.a] Error 1
lipo: no input files specified
lipo: Usage: lipo [input_file] ... [-arch <arch_type> input_file] ... [-info] [-detailed_info] [-output output_file] [-create] [-arch_blank <arch_type>] [-thin <arch_type>] [-remove <arch_type>] ... [-extract <arch_type>] ... [-extract_family <arch_type>] ... [-verify_arch <arch_type> ...] [-replace <arch_type> <file_name>] ...
make[5]: *** [/builds/slave/moz-toolchain/build/stage1/build/tools/clang/runtime/compiler-rt/clang_darwin/ios/libcompiler_rt.a] Error 1
make[4]: *** [BuildRuntimeLibraries] Error 2
rm /builds/slave/moz-toolchain/build/stage1/build/Release/lib/clang/3.5/lib/macho_embedded/.dir /builds/slave/moz-toolchain/build/stage1/build/Release/lib/clang/3.5/lib/darwin/.dir
make[3]: *** [compiler-rt/.makeall] Error 2
make[2]: *** [all] Error 1
make[1]: *** [clang/.makeall] Error 2
make: *** [all] Error 1

Comment 12

[Christian Holler (:decoder)]

@cdiehl, I might need your help to get a Clang Mac package for deploying here.

@rail, could you quickly explain what steps cdiehl would normally have to take to get the build package that we need for deploying?

Thank you both for helping out here :)

If we can't get this working then I'd suggest we proceed with linux64-only for this quarter.

Comment 13

[Rail Aliiev [:rail]]

Per https://wiki.mozilla.org/ReleaseEngineering/How_To/Clang_update "MacOS X builders have everything installed to build clang", not really clear. My guess the requirements are 

* recent xcode
* mercurial
* subversion
* patch

The build instruction can be found on the same page under https://wiki.mozilla.org/ReleaseEngineering/How_To/Clang_update#Compilation (see the last code block in that section + the following paragraph how to find the deliverables).

Hope this helps.

Comment 14

[Christoph Diehl [:posidron]]

Attachment: build-clang-trunk-osx.patch

The attached patch might be not needed on your machines.

% cat build/unix/build-clang/clang.manifest
> [
> {
> "clang_version": "r200213"
> },
> {
> "size": 47,
> "digest": "2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa",
> "algorithm": "sha512",
> "filename": "setup.sh"
> },
> {
> "size": 58997296,
> "digest": "9757d142142442c881b8d1eb31c2fe80e1979a858e6133473b5574a5a3b9cdaf9abed32b2e246b715c9f0eb0969103337918215fc491feae196219e8fb03f0b1",
> "algorithm": "sha512",
> "filename": "clang.tar.bz2"
> }
> ]

https://dl.dropboxusercontent.com/u/54223972/9757d142142442c881b8d1eb31c2fe80e1979a858e6133473b5574a5a3b9cdaf9abed32b2e246b715c9f0eb0969103337918215fc491feae196219e8fb03f0b1

Comment 15

[Christoph Diehl [:posidron]]

(In reply to Christoph Diehl [:cdiehl] from comment #14)

% /Applications/Xcode.app/Contents//Developer/usr/bin/xcodebuild -version
Xcode 5.1
Build version 5B130a

% sw_vers
ProductName:	Mac OS X
ProductVersion:	10.9.2
BuildVersion:	13C64

Comment 16

[Christian Holler (:decoder)]

Thanks Christoph, that was fast! :)

Rail, how should we proceed? Can you upload the build in comment 14? I guess the next step would be to land my reviewed patch, cdiehl's patch (since you also made that change) and the updated manifests for linux64 and osx?

Comment 17

[Rail Aliiev [:rail]]

I uploaded the file. It would be great if you can push the patch to try before we proceed.

Comment 18

[Christian Holler (:decoder)]

Attachment: asan-manifests-200213.patch

Part 1: Final version of the manifests (rolled in the OSX manifest change), keeping f+.

Comment 19

[Christian Holler (:decoder)]

Attachment: clang-build-200213.patch

Part 2: Adjust the build script for Clang trunk on Linux and OSX (merged the two patches). Keeping r+ as the first (Linux) has r+ and the second (OSX) is the same change that rail proposed.

Comment 20

[Christian Holler (:decoder)]

Per comment 17: Linux64 ASan Green on try: https://tbpl.mozilla.org/?tree=Try&rev=17f38b68b370

As discussed on IRC, we don't have a job for OSX yet, so not including that in the try push. Proceeding with landing all patches now.

Comment 21

[Christian Holler (:decoder)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/32f04076b6b5
https://hg.mozilla.org/integration/mozilla-inbound/rev/7dbc40da9c8f

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

This needed builds to be clobbered (all ASAN mochitests on non-clobber builds failed with https://tbpl.mozilla.org/php/getParsedLog.php?id=36896646&tree=Mozilla-Inbound ), so I landed a change to CLOBBER in https://hg.mozilla.org/integration/mozilla-inbound/rev/63dea193e956

Comment 23

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/32f04076b6b5
https://hg.mozilla.org/mozilla-central/rev/7dbc40da9c8f
https://hg.mozilla.org/mozilla-central/rev/63dea193e956