Last Comment Bug 869659 - Cross-compiling JS shell broken since ICU landing
: Cross-compiling JS shell broken since ICU landing
Status: RESOLVED FIXED
[fuzzblocker][leave open]
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: All All
: -- normal (vote)
: ---
Assigned To: Jeff Walden [:Waldo] (remove +bmo to email)
:
: Gregory Szorc [:gps]
Mentors:
Depends on: 912371 1036645
Blocks: 724533 853301
  Show dependency treegraph
 
Reported: 2013-05-07 14:22 PDT by Christian Holler (:decoder)
Modified: 2016-04-05 06:06 PDT (History)
12 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Compile ICU using the CC/CXX determined by Mozilla, not as selected by ICU (766 bytes, patch)
2013-05-10 11:57 PDT, Jeff Walden [:Waldo] (remove +bmo to email)
mh+mozilla: review+
Details | Diff | Splinter Review

Description User image Christian Holler (:decoder) 2013-05-07 14:22:41 PDT
It seems that since ICU landed, I cannot cross-compile JS shells for ARM anymore. I am using this script:


#! /bin/sh
if [ -z $1 ] ; then
    echo "usage: $0 <dirname>"
elif [ -d $1 ] ; then
    echo "directory $1 already exists"
else
    autoconf2.13
    mkdir $1
    cd $1
    ../configure --enable-debug --enable-optimize --with-arch=armv7-a --target=arm-linux-gnueabi
    make --quiet -j 2
fi


When trying to build I get:

intl/icu/lib/libicui18n.a: could not read symbols: File format not recognized
collect2: ld returned 1 exit status
make[1]: *** [libmozjs-23.0a.so] Error 1
make: *** [default] Error 2


I also tried exporting CROSS_COMPILE=1 and setting CC/CXX to the arm variants, but then I'm getting this:


    checking for suffix of executables...
    checking whether we are cross compiling... configure: error: in `/srv/repos/mozilla-central/js/src/dbgarm/intl/icu':
    configure: error: cannot run C compiled programs.
    If you meant to cross compile, use `--host'.
    See `config.log' for more details
    /srv/repos/mozilla-central/js/src/../../intl/icu/source/runConfigureICU: ./configure failed
    ------ config.log ------
    configure:14098: arm-linux-gnueabi-gcc -c  -march=armv7-a -std=gnu99 -fgnu89-inline -ffunction-sections -fdata-sections -pthread -pipe -fprofile-generate -fprofile-correction  conftest.c 1>&5
    configure:14149: checking for correct overload resolution with const and templates
    configure:14184: arm-linux-gnueabi-g++ -c  -march=armv7-a -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -pthread -pipe  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Werror=conversion-null -Wsign-compare -Wno-invalid-offsetof  conftest.C 1>&5
    configure: In function 'int main()':
    configure:14179:37: warning: 'bar' is used uninitialized in this function [-Wuninitialized]
    configure:14210: checking for tm_zone tm_gmtoff in struct tm
    configure:14222: arm-linux-gnueabi-g++ -c  -march=armv7-a -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -pthread -pipe  conftest.C 1>&5
    configure:14264: checking what kind of list files are supported by the linker
    configure:14269: arm-linux-gnueabi-gcc -o conftest.o -c  -march=armv7-a -std=gnu99 -fgnu89-inline -ffunction-sections -fdata-sections -pthread -pipe  conftest.c 1>&5
    configure:14271: arm-linux-gnueabi-gcc -o conftest -lpthread  -Wl,--build-id conftest.list -ldl  1>&5
    configure:14297: checking what kind of ordering can be done with the linker
    configure:14312: arm-linux-gnueabi-gcc -o conftest  -march=armv7-a -std=gnu99 -fgnu89-inline -ffunction-sections -fdata-sections -pthread -pipe  -lpthread  -Wl,--build-id -Wl,--section-ordering-file,conftest.order conftest.c -ldl  1>&5
    /usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/ld: unrecognized option '--section-ordering-file'
    /usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/ld: use the --help option for usage information
    collect2: ld returned 1 exit status
    configure: failed program was:
    #line 14305 "configure"
    #include "confdefs.h"
     
    int main() {
     
    ; return 0; }
    configure:14324: arm-linux-gnueabi-gcc -shared -Wl,-z,defs -Wl,--gc-sections -lpthread  -Wl,--build-id -o libconftest.so -Wl
    configure:14661: checking for posix_fallocate
    configure:14671: arm-linux-gnueabi-gcc -o conftest -Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-unused -march=armv7-a -std=gnu99 -fgnu89-inline -ffunction-sections -fdata-sections -pthread -pipe  -lpthread  -Wl,--build-id conftest.c -ldl  1>&5
    make: *** No targets specified and no makefile found.  Stop.
Comment 1 User image Mike Hommey [:glandium] 2013-05-07 14:57:34 PDT
Quick notes from what I saw at quick glance:
- runConfigureICU doesn't support cross-compilation
- ICU's configure does support it, but you have to pass it a directory containing a native build (--with-cross-build=...), which means we have to do a manual native build first.
Comment 2 User image Mike Hommey [:glandium] 2013-05-08 00:45:59 PDT
And building with CC="gcc -m32" is broken, too. In fact runConfigureICU sets its own CC, so even building with CC="clang" is broken if you don't have gcc. Interestingly, it doesn't set its own CC on OSX...

https://tbpl.mozilla.org/php/getParsedLog.php?id=22721722&tree=Date
Comment 3 User image Ryan VanderMeulen [:RyanVM] 2013-05-08 05:27:47 PDT
Bug 853301 backed out on inbound.
https://hg.mozilla.org/integration/mozilla-inbound/rev/97e73b69c279
Comment 4 User image Norbert Lindenberg 2013-05-08 05:31:39 PDT
For which Mozilla product do you need to cross-compile the JS shell to ARM? Firefox on Android? Firefox OS? If it's just these two, you can use --disable-intl-api for now, and make this bug a blocker for bug 864843 and/or bug 866301.
Comment 5 User image Mike Hommey [:glandium] 2013-05-08 05:36:42 PDT
(In reply to Norbert Lindenberg from comment #4)
> For which Mozilla product do you need to cross-compile the JS shell to ARM?
> Firefox on Android? Firefox OS? If it's just these two, you can use
> --disable-intl-api for now, and make this bug a blocker for bug 864843
> and/or bug 866301.

See bug 853301 comment 13.
Comment 6 User image Mike Hommey [:glandium] 2013-05-08 05:39:04 PDT
Essentially, we need to stop using runConfigureICU and manually call configure with the right variables and arguments for our build. That's the first thing that would solve the 32-bits build on 64-bits toolchain problem. Then there's cross compilation, which requires a double configure-double build setup.
Comment 7 User image Christian Holler (:decoder) 2013-05-08 06:30:53 PDT
(In reply to Norbert Lindenberg from comment #4)
> For which Mozilla product do you need to cross-compile the JS shell to ARM?

It's not a specific Mozilla product, we're just using the JS shell on ARM to test ARM-specific parts of the JS engine more easily. But you're right, we could disable ICU for that purpose.

From comment 6 it sounds like 32 vs. 64 bit is also affected here? We're also building 32 bit shells on our 64 bit servers.
Comment 8 User image Matt Brubeck (:mbrubeck) 2013-05-08 07:23:20 PDT
(In reply to Christian Holler (:decoder) from comment #7)
> From comment 6 it sounds like 32 vs. 64 bit is also affected here? We're
> also building 32 bit shells on our 64 bit servers.

And we plan to switch to building Desktop Firefox for 32-bit Linux on 64-bit builders soon (bug 857697).  This may become urgent if memory to link libxul on Linux keeps rising.
Comment 9 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-05-08 10:47:00 PDT
I guess I'll take this, although if the going gets too interesting I may punt to someone who actually understands our build and cross-compilation code.
Comment 10 User image Mike Hommey [:glandium] 2013-05-09 03:51:26 PDT
So in fact, for the 32-bits build on 64-bits toolchain problem, adding CC="$CC" CXX="$CXX" to the runConfigureICU command should be enough, because I misread, it's not overriding CC/CXX, it's just not inheriting it (and ends up using clang if it's installed, even when the rest uses gcc...).

Arguably, we may want to add some or all of our build flags, too (like, optimization, pgo, -ffunction-section -fdata-section, etc.).
Comment 11 User image Gregory Szorc [:gps] 2013-05-10 10:07:38 PDT
Are we going to let ICU slip to 24 because of this? ICU blew up the weekend before the 22 train and I don't want to see a repeat.

What are we going to do about this?
Comment 12 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-05-10 10:36:52 PDT
Norbert was contracting, and his contract ended, so you want me now for this.  :-)

I was working on importing test262 into the tree to get more JS test coverage as my biggest priority.  But the clock does argue for dropping that for at least a few days to pick this up.  I'll see what I can do before uplift.
Comment 13 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-05-10 11:57:48 PDT
Created attachment 748125 [details] [diff] [review]
Compile ICU using the CC/CXX determined by Mozilla, not as selected by ICU

The 64-to-32 Linux compilation steps on https://developer.mozilla.org/en-US/docs/SpiderMonkey/Build_Documentation work with this, so this is at least a partial improvement.  Is more needed?
Comment 14 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-05-10 11:58:53 PDT
Comment on attachment 748125 [details] [diff] [review]
Compile ICU using the CC/CXX determined by Mozilla, not as selected by ICU

Does this work for the fuzzers, or is more needed?
Comment 15 User image Christian Holler (:decoder) 2013-05-10 11:59:57 PDT
(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #14)

> Does this work for the fuzzers, or is more needed?

I really need cross compiling to ARM, otherwise it's going to get really hard to keep fuzzing the JS shell in my qemu instances. Compiling in the qemu instance natively is going to be extremely slow.
Comment 16 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-05-10 14:23:08 PDT
(In reply to Christian Holler (:decoder) from comment #15)
> I really need cross compiling to ARM

Sure.  Does this patch work for that?  I only mentioned 64-to-32 because that's the easiest cross-compiling setup for me to test -- I'd naively expect that would also fix ARM cross-compiling, but I really don't know.  :-)  Ergo the request to try and see if it does.
Comment 17 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-05-10 17:38:36 PDT
Apparently not.  And apparently 64-to-32 ain't really cross-compiling, either.  Working on setting up an ARM toolchain now to debug here, but anyone who knows anything about cross-compiling will probably make faster progress than I will.  :-)
Comment 18 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-05-10 17:44:17 PDT
Or, no.  ICU apparently has to be built for the host system, then built for the target system, to cross-compile.  Looking into this now:

http://source.icu-project.org/repos/icu/icu/trunk/readme.html#HowToCrossCompileICU
Comment 19 User image Christian Holler (:decoder) 2013-05-10 21:58:00 PDT
I figured out that with --disable-intl-api I'm able to cross-compile for ARM, so that's at least a way to make this less of a blocker for now :) It would still be good to resolve it, but for now it should be good with that option for ARM testing.
Comment 20 User image Mike Hommey [:glandium] 2013-05-10 22:30:11 PDT
Comment on attachment 748125 [details] [diff] [review]
Compile ICU using the CC/CXX determined by Mozilla, not as selected by ICU

Review of attachment 748125 [details] [diff] [review]:
-----------------------------------------------------------------

This should enough for 32-on-64 builds as mentioned in comment 10. You can test by relanding bug 853301 along with this patch on the "date" project branch.
Comment 21 User image Mike Hommey [:glandium] 2013-05-10 22:31:54 PDT
(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #18)
> Or, no.  ICU apparently has to be built for the host system, then built for
> the target system, to cross-compile.  Looking into this now:
> 
> http://source.icu-project.org/repos/icu/icu/trunk/readme.
> html#HowToCrossCompileICU

See comment 6.

(In reply to Mike Hommey [:glandium] from comment #20)
> This should enough for 32-on-64 builds as mentioned in comment 10. You can
> test by relanding bug 853301 along with this patch on the "date" project
> branch.

That being said, bug 853301 can't reland as-is. A new patch will have to be written to make the change in configure instead of mozconfig.
Comment 22 User image Christian Holler (:decoder) 2013-06-05 04:21:58 PDT
Comment on attachment 748125 [details] [diff] [review]
Compile ICU using the CC/CXX determined by Mozilla, not as selected by ICU

Canceling feedback here, as this doesn't resolve the cross-compilation issues.
Comment 23 User image Jeff Walden [:Waldo] (remove +bmo to email) 2013-06-26 14:51:26 PDT
Landed the use-CC/CXX patch, spinning up a mobile build environment now to continue work here:

https://hg.mozilla.org/integration/mozilla-inbound/rev/ddccec687149
Comment 24 User image Ed Morley [:emorley] 2013-06-27 03:50:25 PDT
https://hg.mozilla.org/mozilla-central/rev/ddccec687149
Comment 25 User image Ryan VanderMeulen [:RyanVM] 2013-07-02 12:35:08 PDT
https://hg.mozilla.org/mozilla-central/rev/ecdfb39891a7
Comment 26 User image Jan Beich 2013-07-31 00:17:37 PDT
Comment on attachment 748125 [details] [diff] [review]
Compile ICU using the CC/CXX determined by Mozilla, not as selected by ICU

Doesn't help on a system without gcc.

$ /m/js/src/configure
...
export CPP= CC=gcc CXX=g++ CPPFLAGS= -DU_USING_ICU_NAMESPACE=0 -DU_NO_DEFAULT_INCLUDE_UTF_HEADERS=1 -DUCONFIG_NO_LEGACY_CONVERSION -DUCONFIG_NO_TRANSLITERATION -DUCONFIG_NO_REGULAR_EXPRESSIONS -DUCONFIG_NO_BREAK_ITERATION CFLAGS=-fPIC -O3 CXXFLAGS=-fPIC -O3 LDFLAGS= MAKE=
Running ./configure  --enable-static --disable-shared --enable-extras=no --enable-icuio=no --enable-layout=no --enable-tests=no --enable-samples=no for BSD using the GNU C++ compiler

checking for ICU version numbers... release 50.1.2, library 50.1.2, unicode version 6.2
checking build system type... x86_64-unknown-freebsd10.0
checking host system type... x86_64-unknown-freebsd10.0
checking whether to build debug libraries... no
checking whether to build release libraries... yes
checking whether the C compiler works... no
configure: error: in `/cache/m/js/src/intl/icu':
configure: error: C compiler cannot create executables
See `config.log' for more details
/m/js/src/../../intl/icu/source/runConfigureICU: ./configure failed
------ config.log ------
configure:14348:37: note: initialize the variable 'bar' to silence this warning
                      const int* bar;
                                    ^
                                     = nullptr
1 warning generated.
configure:14380: checking for tm_zone tm_gmtoff in struct tm
configure:14392: c++ -c  -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pipe -Qunused-arguments   -I/usr/local/include conftest.C 1>&5
configure:14434: checking what kind of list files are supported by the linker
configure:14439: cc -o conftest.o -c  -std=gnu99 -fgnu89-inline -ffunction-sections -fdata-sections -pipe -Qunused-arguments   -I/usr/local/include conftest.c 1>&5
configure:14441: cc -o conftest -pthread  -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id conftest.list  1>&5
configure:14467: checking what kind of ordering can be done with the linker
configure:14482: cc -o conftest  -std=gnu99 -fgnu89-inline -ffunction-sections -fdata-sections -pipe -Qunused-arguments   -I/usr/local/include -pthread  -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -Wl,--section-ordering-file,conftest.order conftest.c  1>&5
/usr/local/bin/ld: unrecognized option '--section-ordering-file'
/usr/local/bin/ld: use the --help option for usage information
cc: error: linker command failed with exit code 1 (use -v to see invocation)
configure: failed program was:
#line 14475 "configure"
#include "confdefs.h"

int main() {

; return 0; }
configure:14494: cc -shared -Wl,-z,defs -pthread  -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -o libconftest.so -Wl
configure:14831: checking for posix_fallocate
configure:14841: cc -o conftest -Qunused-arguments -Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Werror=implicit-function-declaration -Wtype-limits -Wempty-body -Wsign-compare -Wno-unused -std=gnu99 -fgnu89-inline -ffunction-sections -fdata-sections -pipe -Qunused-arguments   -I/usr/local/include -pthread  -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id conftest.c  1>&5
zsh: exit 1
Comment 27 User image Jan Beich 2013-08-14 17:55:22 PDT
Nevermind. A few platforms in runConfigureICU don't inherit CC/CXX
from js/src/configure. I've solved my error with attachment 790417 [details] [diff] [review].
Comment 28 User image Makoto Kato [:m_kato] 2013-12-01 18:47:05 PST
decoder, although I think that this is fixed by bug 912371 except to Android and Gonk, does this bug still occur?
Comment 29 User image Ted Mielczarek [:ted.mielczarek] 2016-04-05 06:06:10 PDT
I believe bug 1036645 fixed this.

Note You need to log in before you can comment on or make changes to this bug.