756816 - android-x86 builds fail on build slaves because x86 GCC is linked against a too-new glibc

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[(not currently active) Ted Mielczarek]

I pushed a test android-x86 build to try:
https://hg.mozilla.org/try/rev/d5fa36c475bd

and it failed because the android-x86 GCC won't run because it's linked to a glibc that's newer than what we have on the build slaves:
configure:2046: checking for /tools/android-ndk-r7b/toolchains/x86-4.4.3/prebuilt/linux-x86/bin/i686
-android-linux-gcc
configure:2080: checking for gcc
configure:2193: checking whether the C compiler (/tools/android-ndk-r7b/toolchains/x86-4.4.3/prebuil
t/linux-x86/bin/i686-android-linux-gcc -mandroid -fno-short-enums -fno-exceptions  -mandroid -L/tool
s/android-ndk-r7b/platforms/android-9/arch-x86/usr/lib -Wl,-rpath-link=/tools/android-ndk-r7b/platfo
rms/android-9/arch-x86/usr/lib --sysroot=/tools/android-ndk-r7b/platforms/android-9/arch-x86 -llog -
Wl,--allow-shlib-undefined ) works
configure:2209: /tools/android-ndk-r7b/toolchains/x86-4.4.3/prebuilt/linux-x86/bin/i686-android-linu
x-gcc -o conftest -mandroid -fno-short-enums -fno-exceptions  -isystem /tools/android-ndk-r7b/platfo
rms/android-9/arch-x86/usr/include  -mandroid -L/tools/android-ndk-r7b/platforms/android-9/arch-x86/
usr/lib -Wl,-rpath-link=/tools/android-ndk-r7b/platforms/android-9/arch-x86/usr/lib --sysroot=/tools
/android-ndk-r7b/platforms/android-9/arch-x86 -llog -Wl,--allow-shlib-undefined  conftest.c  1>&5
/tools/android-ndk-r7b/toolchains/x86-4.4.3/prebuilt/linux-x86/bin/i686-android-linux-gcc: /lib/libc
.so.6: version `GLIBC_2.11' not found (required by /tools/android-ndk-r7b/toolchains/x86-4.4.3/prebu
ilt/linux-x86/bin/i686-android-linux-gcc)
configure: failed program was:

#line 2204 "configure"
#include "confdefs.h"

main(){return(0);}

We'll need to figure out a way around this to get Android-x86 builds on the slaves.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(adding jhford, as he did a bunch of the work here, in case I missed anything).

From looking at https://wiki.mozilla.org/Mobile/Fennec/Android#Install_Android_NDK, we already deployed new NDK "r7c", and new compiler/linker as part of bug#675572. (Just noticed that wiki mentions r7c, while bug mentions r7b.)

Is it possible the errors in comment#0 are actually mozconfig errors?

Comment 2

[(not currently active) Ted Mielczarek]

The exact same mozconfig works fine for me with the exact same NDK on my local Ubuntu 10.04 machine.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

I would need access to one of these machines to debug this issue because it is very specific to the machine's configuration.  We only have ancient versions of glibc available on the build slaves.  We might have a newer version of glibc available in one of our custom toolchains, I wonder if we could export an LD_LIBRARY_PATH to allow the android toolchain to dynamically link against our custom gcc-4.5 toolchain's glibc.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

I looked around on one of these machines and couldn't find any other glibc installations that do have the GLIBC symbol.  I tried building an alternate glibc but it didn't work.  The options I see are to upgrade the centos5 build machines to have a newer glibc or to use Mock.

It wouldn't be too hard to use the b2g build environment as a basis, with the android ndk and sdk packages in the mozilla repo (and installed) to get a basic build environment.

Comment 5

[Mike Hommey [:glandium]]

We should probably use mock.

Comment 6

[Brad Lassey [:blassey] (use needinfo?)]

John, for the ndk r7c, did you rebuild it from source? If so, any chance you linked it against the older version of ndk that is on our builders?

Comment 7

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Brad Lassey [:blassey] from comment #6)
> John, for the ndk r7c, did you rebuild it from source? If so, any chance you
> linked it against the older version of ndk that is on our builders?

That was a question for John Ford

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

We only rebuild the specific arm gcc-4.6.3 toolchain.  We just repackage the other parts, including the x86 toolchain.

Comment 9

[(not currently active) Ted Mielczarek]

I'm going to re-test with a Try build since we've moved our Android builds to the mock slaves.

Comment 10

[(not currently active) Ted Mielczarek]

https://tbpl.mozilla.org/?tree=Try&rev=5fd4e2336e7d

Comment 11

[(not currently active) Ted Mielczarek]

I can't tell if this is fixed because the mock slaves only install an older NDK,and not the r7b we need to build android-x86.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

We currently have green android x86 builds visible because of the work done in bug#750366.

Is there anything left to do here?

Comment 13

[(not currently active) Ted Mielczarek]

No, this was fixed by the move to mock, it was just hard to test until someone did the work to fixup the mock build configs. (Thanks Kim!)