1206193 - ICU configure failure on Windows ASan builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(Away)]

1:28.64 intl\icu\target> checking whether the C compiler works... no
 1:28.64 intl\icu\target> configure: error: in `/usr/s/clangcl/obj/asan/intl/icu/target':
 1:28.64 intl\icu\target> configure: error: C compiler cannot create executables
 1:28.64 intl\icu\target> See `config.log' for more details
 
 config.log contains:
 
 configure:2962: checking whether the C compiler works
configure:2984: clang-cl  -Qunused-arguments -fsanitize=address -Z7 -fms-compatibility-version=18.00.30723 -fallback -TC -nologo -D_HAS_EXCEPTIONS=0 -W3 -Gy -FS -wd4244 -wd4267 -wd4819 -we4553 -Zi -UDEBUG -DNDEBUG -O1 -Oi -MD  -DU_USING_ICU_NAMESPACE=0 -DU_NO_DEFAULT_INCLUDE_UTF_HEADERS=1 -DUCONFIG_NO_LEGACY_CONVERSION -DUCONFIG_NO_TRANSLITERATION -DUCONFIG_NO_REGULAR_EXPRESSIONS -DUCONFIG_NO_BREAK_ITERATION -DUCONFIG_NO_IDNA -DU_CHARSET_IS_UTF8 -I/usr/s/clangcl/intl/icu/source/common -I/usr/s/clangcl/intl/icu/source/i18n -Qunused-arguments  -DEBUG -DEBUGTYPE:CV clang_rt.asan_dynamic-i386.lib clang_rt.asan_dynamic_runtime_thunk-i386.lib -DYNAMICBASE conftest.c  >&5
In file included from <built-in>:297:
<command line>(15,17) :  warning(clang): ISO C99 requires whitespace after the macro name [-Wc99-extensions]
#define EBUGTYPE:CV 1
                ^
1 warning generated.
configure:2988: $? = 0
configure:3026: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "ICU"
| #define PACKAGE_TARNAME "International Components for Unicode"
| #define PACKAGE_VERSION "55.1"
| #define PACKAGE_STRING "ICU 55.1"
| #define PACKAGE_BUGREPORT "http://icu-project.org/bugs"
| #define PACKAGE_URL "http://icu-project.org"
| /* end confdefs.h.  */
| 
| int
| main ()
| {
| 
|   ;
|   return 0;
| }
configure:3031: error: in `/usr/s/clangcl/obj/asan/intl/icu/target':
configure:3033: error: C compiler cannot create executables
See `config.log' for more details

Comment 1

[(Away)]

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj/asan
ac_add_options --enable-warnings-as-errors
# ac_add_options --without-intl-api

export CC=clang-cl
export CXX=clang-cl
export CFLAGS="-fsanitize=address -Z7"
export CXXFLAGS="-fsanitize=address -Z7"

ac_add_options --enable-address-sanitizer
ac_add_options --disable-jemalloc
export LDFLAGS="clang_rt.asan_dynamic-i386.lib clang_rt.asan_dynamic_runtime_thunk-i386.lib"
export HOST_CFLAGS=" "
export HOST_CXXFLAGS=" "
export HOST_LDFLAGS=" "

Comment 2

[(Away)]

The root of this problem appears to be that we're passing linker flags to the compiler:
-DEBUG -DEBUGTYPE:CV clang_rt.asan_dynamic-i386.lib clang_rt.asan_dynamic_runtime_thunk-i386.lib -DYNAMICBASE

-DEBUG gets interpreted as "#define EBUG" which is weird but benign.
-DEBUGTYPE:CV gets interpreted as "#define EBUGTYPE:CV" which causes the error above.

Glandium suggested that I remove -DEBUGTYPE:CV since it's an ancient flag that defaults to the right thing nowadays, but removing it wasn't enough, since the "clang_rt.asan_dynamic-i386.lib" parameter still confuses the compiler.

Comment 3

[(no longer active)]

The cl driver is pretty permissive in handling arguments that it does not understand, but clang-cl doesn't accept those by default, and in fact if this was not being interpreted as -D, you'd get a compiler error about the unrecognized flag.

Removing -DEBUGTYPE:CV makes sense to me.

About the .lib issues, I have previously fixed the clang-cl driver to pass those correctly to the linker.  It's possible that I have missed something there.  Is there a minimal test case for that part of the bug?  (Please specify where the .lib files are on your disk relative to the source file and the compiler.  Also it would be helpful if you include the -### output.  Thanks!)

Comment 4

[(Away)]

My memory is a little fuzzy since I encountered this a while ago and forgot to file the bug, but.

I think the lib problems are not of the sort that you have in mind. I think what happened is clang-cl thought it was being asked to produce clang_rt.asan_dynamic-i386.exe, so configure didn't find the conftest.exe that it was looking for.

If that's not helpful then I can re-run the experiment and paste actual output when I have time.

Comment 5

[(no longer active)]

Hmm, that is not at all what I would expect from what the code tries to do, and unfortunately I have no idea what would cause such a bug without seeing a test case.  I'd appreciate if you could get some precise STRs here.  Thanks!

Comment 6

[(Away)]

My STR are:
- build clang-cl and asan, have them on your path
- set your mozconfig to comment 1
- mach configure

Let me know if you need more...

Comment 7

[(no longer active)]

I was hoping for something like:

echo 'int main() { return 0; }' > test.cpp
clang-cl <args that break the build go here> test.cpp

:-)

Comment 8

[(Away)]

Well, you could look at the command line from comment 0, but reproducing that is not really "interesting". The real question is why we ran that command in the first place. This is a build system issue.

Comment 9

[(Away)]

The core problem seems to be that we are passing the mozconfig's LDFLAGS to the compiler (not linker). This would fail with cl as well (actually, it fails more loudly):

$ echo 'int main() { return 0; }' > test.cpp

$ ls
test.cpp

$ cl  -Qunused-arguments -fsanitize=address -Z7 -fms-compatibility-version=18.0
0.30723 -fallback -TC -nologo -D_HAS_EXCEPTIONS=0 -W3 -Gy -FS -O1 -Oi -MD  -DEB
UG -DEBUGTYPE:CV clang_rt.asan_dynamic-i386.lib clang_rt.asan_dynamic_runtime_t
hunk-i386.lib -DYNAMICBASE test.cpp
cl : Command line warning D9002 : ignoring unknown option '-Qunused-arguments'
cl : Command line warning D9002 : ignoring unknown option '-fsanitize=address'
cl : Command line warning D9002 : ignoring unknown option '-fms-compatibility-ve
rsion=18.00.30723'
cl : Command line warning D9002 : ignoring unknown option '-fallback'
clang_rt.asan_dynamic-i386.lib
c1 : fatal error C1083: Cannot open source file: 'clang_rt.asan_dynamic-i386.lib
': No such file or directory
clang_rt.asan_dynamic_runtime_thunk-i386.lib
c1 : fatal error C1083: Cannot open source file: 'clang_rt.asan_dynamic_runtime_
thunk-i386.lib': No such file or directory
test.cpp
Generating Code...

$ ls
test.cpp  test.obj

$ rm test.obj

$ clang-cl  -Qunused-arguments -fsanitize=address -Z7 -fms-compatibility-versio
n=18.00.30723 -fallback -TC -nologo -D_HAS_EXCEPTIONS=0 -W3 -Gy -FS -O1 -Oi -MD
  -DEBUG -DEBUGTYPE:CV clang_rt.asan_dynamic-i386.lib clang_rt.asan_dynamic_run
time_thunk-i386.lib -DYNAMICBASE test.cpp
In file included from <built-in>:297:
<command line>(5,17) :  warning(clang): ISO C99 requires whitespace after the
      macro name [-Wc99-extensions]
#define EBUGTYPE:CV 1
                ^
1 warning generated.

$ ls
clang_rt.asan_dynamic-i386.exe  clang_rt.asan_dynamic-i386.pdb  test.cpp

Comment 10

[(no longer active)]

Hmm, I think this is http://llvm.org/PR20923 which should be fixed.

Have you verified that clang_rt.asan_dynamic-i386.lib exists in a directory in $LIB (or in the current working directory)?

Comment 11

[(Away)]

> Have you verified that clang_rt.asan_dynamic-i386.lib exists in a directory
> in $LIB (or in the current working directory)?

Yes, it exists in one of my $LIB directories.

Comment 12

[Mike Hommey [:glandium]]

How does it go if you put the .lib file names *after* test.cpp on the command line ?

Comment 13

[(Away)]

(In reply to Mike Hommey [:glandium] from comment #12)
> How does it go if you put the .lib file names *after* test.cpp on the
> command line ?

That seems to work for clang-cl, but cl still complains about Cannot open source file: 'clang_rt.asan_dynamic-i386.lib'.

Comment 14

[(no longer active)]

OK, I can reproduce this on clang-cl.  Investigating...

Comment 15

[Mike Hommey [:glandium]]

(In reply to David Major [:dmajor] from comment #13)
> (In reply to Mike Hommey [:glandium] from comment #12)
> > How does it go if you put the .lib file names *after* test.cpp on the
> > command line ?
> 
> That seems to work for clang-cl, but cl still complains about Cannot open
> source file: 'clang_rt.asan_dynamic-i386.lib'.

You lost me here. Why would cl's behavior matter?

Comment 16

[(Away)]

> Why would cl's behavior matter?

clang-cl is intended to behave like cl. At some point in the future, clang-cl may start matching cl's behavior on this, and we'd be back to being busted.

Comment 17

[Mike Hommey [:glandium]]

sure, but if moving clang_rt.asan_dynamic-i386.lib last works, then you can make it -link clang_rt.asan_dynamic-i386.lib and make cl happy.

Comment 18

[(no longer active)]

(In reply to Mike Hommey [:glandium] from comment #17)
> sure, but if moving clang_rt.asan_dynamic-i386.lib last works, then you can
> make it -link clang_rt.asan_dynamic-i386.lib and make cl happy.

If we can do that, then that is preferable.  My patch for fixing this in clang is turning into a huge mess.  It seems like clang's job construction code has hardcoded assumptions about the first argument to the linker always being used to derive the name of the output executable, so I am not in a rush to fix it in clang if we can easily work around it on our side.  :-)

Comment 19

[(Away)]

Ehsan, please do not spend any more time on this while there are still much more important clang bugs outstanding :-) I kind of regret that this bug has gone on even this far.

Comment 20

[Mike Hommey [:glandium]]

Can anyone confirm that comment 17 works?

Comment 21

[(not currently active) Ted Mielczarek]

bug 1239083 should make this moot by not running ICU's configure anymore.