691071 - debug build of android crash on tegra during startup

Status: RESOLVED INVALID

(Firefox for Android Graveyard :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Taking the latest debug build of android produced from mozilla-central, I run into a crash while starting up the test:

I/ActivityManager( 1021): Starting activity: Intent { act=android.intent.action.VIEW dat=http://192.168.1.109:8888/jsreftest/tests/jstests.list flg=0x10000000 pkg=org.mozilla.fennec cmp=org.mozilla.fennec/.App (has extras) }
I/ActivityManager( 1021): Start proc org.mozilla.fennec for activity org.mozilla.fennec/.App: pid=4399 uid=10034 gids={3003, 1015, 1007, 1006}
D/dalvikvm( 1448): GC_EXPLICIT freed 1398 objects / 162776 bytes in 95ms
I/wpa_supplicant( 1335): CTRL-EVENT-STATE-CHANGE id=-1 state=1
V/WifiMonitor( 1021): Event [CTRL-EVENT-STATE-CHANGE id=-1 state=1]
V/WifiStateTracker( 1021): Changing supplicant state: SCANNING ==> INACTIVE
D/NetworkStateTracker( 1021): setDetailed state, old =SCANNING and new state=IDLE
D/ConnectivityService( 1021): Dropping ConnectivityChange for WIFI: DISCONNECTED/IDLE
D/dalvikvm( 1448): GC_EXPLICIT freed 344 objects / 53872 bytes in 75ms
D/dalvikvm( 4399): GC_FOR_MALLOC freed 2023 objects / 151456 bytes in 36ms
I/GeckoApp( 4399): create
I/GeckoApp( 4399): Checking for an update
I/GeckoApp( 4399): start
I/dalvikvm( 4399): Could not find method android.hardware.Camera.getNumberOfCameras, referenced from method org.mozilla.gecko.GeckoAppShell.initCamera
W/dalvikvm( 4399): VFY: unable to resolve static method 129: Landroid/hardware/Camera;.getNumberOfCameras ()I
D/dalvikvm( 4399): VFY: replacing opcode 0x71 at 0x004b
I/dalvikvm( 4399): Could not find method android.hardware.Camera.open, referenced from method org.mozilla.gecko.GeckoAppShell.initCamera
W/dalvikvm( 4399): VFY: unable to resolve static method 131: Landroid/hardware/Camera;.open (I)Landroid/hardware/Camera;
D/dalvikvm( 4399): VFY: replacing opcode 0x71 at 0x0053
D/dalvikvm( 4399): VFY: dead code 0x004e-0051 in Lorg/mozilla/gecko/GeckoAppShell;.initCamera (Ljava/lang/String;III)[I
D/dalvikvm( 4399): VFY: dead code 0x0056-0176 in Lorg/mozilla/gecko/GeckoAppShell;.initCamera (Ljava/lang/String;III)[I
D/dalvikvm( 4399): VFY: dead code 0x0190-0192 in Lorg/mozilla/gecko/GeckoAppShell;.initCamera (Ljava/lang/String;III)[I
D/dalvikvm( 4399): Trying to load lib /data/data/org.mozilla.fennec/lib/libmozutils.so 0x4430a418
D/dalvikvm( 4399): Added shared lib /data/data/org.mozilla.fennec/lib/libmozutils.so 0x4430a418
D/dalvikvm( 4399): No JNI_OnLoad found in /data/data/org.mozilla.fennec/lib/libmozutils.so 0x4430a418, skipping init
I/GeckoAppShell( 4399): env0: ANDROID_SOCKET_zygote=10
I/GeckoAppShell( 4399): env1: ANDROID_BOOTLOGO=1
I/GeckoAppShell( 4399): env2: EXTERNAL_STORAGE=/mnt/sdcard
I/GeckoAppShell( 4399): env3: ANDROID_ASSETS=/system/app
I/GeckoAppShell( 4399): env4: ASEC_MOUNTPOINT=/mnt/asec
I/GeckoAppShell( 4399): env5: PATH=/sbin:/system/sbin:/system/bin:/system/xbin
I/GeckoAppShell( 4399): env6: ANDROID_DATA=/data
I/GeckoAppShell( 4399): env7: BOOTCLASSPATH=/system/framework/core.jar:/system/framework/ext.jar:/system/framework/framework.jar:/system/framework/android.policy.jar:/system/framework/services.jar:/system/framework/com.nvidia.graphics.jar
I/GeckoAppShell( 4399): env8: ANDROID_PROPERTY_WORKSPACE=9,32768
I/GeckoAppShell( 4399): env9: ANDROID_ROOT=/system
I/GeckoAppShell( 4399): env10: LD_LIBRARY_PATH=/system/lib
I/GeckoAppShell( 4399): env11: MOZ_CRASHREPORTER_NO_REPORT=1
I/GeckoAppShell( 4399): env12: MOZ_CRASHREPORTER=1
I/GeckoAppShell( 4399): env13: XPCOM_MEM_BLOAT_LOG=/tmp/tmpvNNtBm/runreftest_leaks.log
I/GeckoAppShell( 4399): env14: XPCOM_DEBUG_BREAK=stack
I/GeckoAppShell( 4399): env15: MOZ_PROCESS_LOG=/tmp/tmpuRQVJPpidlog
I/GeckoAppShell( 4399): env16: NO_EM_RESTART=1
I/GeckoAppShell( 4399): env17: null
I/GeckoApp( 4399): resume
W/InputManagerService( 1021): Starting input on non-focused client com.android.internal.view.IInputMethodClient$Stub$Proxy@4450a838 (uid=10031 pid=1448)
I/GeckoSurfaceView( 4399): surface created
I/GeckoSurfaceView( 4399): surfaceChanged: fmt: -1 dim: 1600 1175
I/ActivityManager( 1021): Displayed activity org.mozilla.fennec/.App: 450 ms (total 450 ms)
E/GeckoLibLoad( 4399): resolved libmozalloc.so to /data/data/org.mozilla.fennec/cache/libmozalloc.so
E/GeckoLibLoad( 4399): libmozalloc.so: spent 16930
E/GeckoLibLoad( 4399): resolved libnspr4.so to /data/data/org.mozilla.fennec/cache/libnspr4.so
E/GeckoLibLoad( 4399): libnspr4.so: spent 38936
E/GeckoLibLoad( 4399): resolved libplc4.so to /data/data/org.mozilla.fennec/cache/libplc4.so
E/GeckoLibLoad( 4399): libplc4.so: spent 27119
E/GeckoLibLoad( 4399): resolved libplds4.so to /data/data/org.mozilla.fennec/cache/libplds4.so
E/GeckoLibLoad( 4399): libplds4.so: spent 22732
E/GeckoLibLoad( 4399): resolved libmozsqlite3.so to /data/data/org.mozilla.fennec/cache/libmozsqlite3.so
E/GeckoLibLoad( 4399): libmozsqlite3.so: spent 247655
E/GeckoLibLoad( 4399): resolved libnssutil3.so to /data/data/org.mozilla.fennec/cache/libnssutil3.so
E/GeckoLibLoad( 4399): libnssutil3.so: spent 55698
E/GeckoLibLoad( 4399): resolved libnss3.so to /data/data/org.mozilla.fennec/cache/libnss3.so
E/GeckoLibLoad( 4399): Couldn't load /data/data/org.mozilla.fennec/cache/libnss3.so because Cannot load library: link_image[2046]: failed to link libnss3.so
E/GeckoLibLoad( 4399): libnss3.so: spent 276113
E/GeckoLibLoad( 4399): resolved libssl3.so to /data/data/org.mozilla.fennec/cache/libssl3.so
E/GeckoLibLoad( 4399): Couldn't load /data/data/org.mozilla.fennec/cache/libssl3.so because Cannot load library: link_image[2046]: failed to link libssl3.so
E/GeckoLibLoad( 4399): libssl3.so: spent 93572
E/GeckoLibLoad( 4399): resolved libsmime3.so to /data/data/org.mozilla.fennec/cache/libsmime3.so
E/GeckoLibLoad( 4399): Couldn't load /data/data/org.mozilla.fennec/cache/libsmime3.so because Cannot load library: link_image[2046]: failed to link libsmime3.so
E/GeckoLibLoad( 4399): libsmime3.so: spent 58890
E/GeckoLibLoad( 4399): resolved libxul.so to /data/data/org.mozilla.fennec/cache/libxul.so
E/GeckoLibLoad( 4399): Couldn't load /data/data/org.mozilla.fennec/cache/libxul.so because Cannot load library: link_image[2046]: failed to link libxul.so
E/GeckoLibLoad( 4399): libxul.so: spent 4786889
E/GeckoLibLoad( 4399): resolved libxpcom.so to /data/data/org.mozilla.fennec/cache/libxpcom.so
E/GeckoLibLoad( 4399): Couldn't load /data/data/org.mozilla.fennec/cache/libxpcom.so because Cannot load library: link_image[2046]: failed to link libxpcom.so
E/GeckoLibLoad( 4399): libxpcom.so: spent 18018
E/GeckoLibLoad( 4399): resolved libnssckbi.so to /data/data/org.mozilla.fennec/cache/libnssckbi.so
E/GeckoLibLoad( 4399): libnssckbi.so: spent 176333
E/GeckoLibLoad( 4399): resolved libfreebl3.so to /data/data/org.mozilla.fennec/cache/libfreebl3.so
E/GeckoLibLoad( 4399): libfreebl3.so: spent 56058
E/GeckoLibLoad( 4399): resolved libsoftokn3.so to /data/data/org.mozilla.fennec/cache/libsoftokn3.so
E/GeckoLibLoad( 4399): libsoftokn3.so: spent 81144
E/GeckoLibLoad( 4399): Couldn't get a handle to libxul!
E/GeckoLibLoad( 4399): Loaded libs in 5959ms total, 700ms user, 510ms system, 22 faults
I/DEBUG   (  937): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG   (  937): Build fingerprint: 'nvidia/harmony/harmony/harmony:2.2/FRF91/20101105.183900:eng/test-keys'
I/DEBUG   (  937): pid: 4399, tid: 4409  >>> org.mozilla.fennec <<<
I/DEBUG   (  937): signal 11 (SIGSEGV), fault addr 00000000
I/DEBUG   (  937):  r0 00205048  r1 44384520  r2 42944fc4  r3 00000000
I/DEBUG   (  937):  r4 47c97b70  r5 00000000  r6 44384520  r7 47c97b40
I/DEBUG   (  937):  r8 47c97b50  r9 42944f84  10 42944f70  fp 00221918
I/DEBUG   (  937):  ip 80c19361  sp 47c97b40  lr 80c19377  pc 00000000  cpsr a0000010
I/DEBUG   (  937):  d0  0000005042a00000  d1  0000000042a00000
I/DEBUG   (  937):  d2  3f8000003f800000  d3  c16da00040718000
I/DEBUG   (  937):  d4  4113200000000000  d5  000000004bb50000
I/DEBUG   (  937):  d6  c48ee00000093200  d7  3eaaaaab3f800000
I/DEBUG   (  937):  d8  0000000000000000  d9  0000000000000000
I/DEBUG   (  937):  d10 0000000000000000  d11 0000000000000000
I/DEBUG   (  937):  d12 0000000000000000  d13 0000000000000000
I/DEBUG   (  937):  d14 0000000000000000  d15 0000000000000000
I/DEBUG   (  937):  scr 80000012
I/DEBUG   (  937): 
I/DEBUG   (  937):          #00  pc 00000000  
I/DEBUG   (  937):          #01  lr 80c19377  /data/data/org.mozilla.fennec/lib/libmozutils.so
I/DEBUG   (  937): 
I/DEBUG   (  937): code around pc:
I/DEBUG   (  937): 
I/DEBUG   (  937): code around lr:
I/DEBUG   (  937): 80c19354 46180300 0718f107 bd8046bd b082b580 
I/DEBUG   (  937): 80c19364 6078af00 4b056039 681b447b 68396878 
I/DEBUG   (  937): 80c19374 f1074798 46bd0708 bf00bd80 00010e5c 
I/DEBUG   (  937): 80c19384 b084b580 60f8af00 607a60b9 447b4b05 
I/DEBUG   (  937): 80c19394 68f8681b 687a68b9 f1074798 46bd0710 
I/DEBUG   (  937): 
I/DEBUG   (  937): stack:
I/DEBUG   (  937):     47c97b00  00000000  
I/DEBUG   (  937):     47c97b04  afd10500  /system/lib/libc.so
I/DEBUG   (  937):     47c97b08  afd43728  /system/lib/libc.so
I/DEBUG   (  937):     47c97b0c  0021daa0  [heap]
I/DEBUG   (  937):     47c97b10  0024d958  [heap]
I/DEBUG   (  937):     47c97b14  0000a000  [heap]
I/DEBUG   (  937):     47c97b18  80816fc0  /system/lib/libdvm.so
I/DEBUG   (  937):     47c97b1c  00221918  [heap]
I/DEBUG   (  937):     47c97b20  42944f70  
I/DEBUG   (  937):     47c97b24  afd0be23  /system/lib/libc.so
I/DEBUG   (  937):     47c97b28  0021da70  [heap]
I/DEBUG   (  937):     47c97b2c  0022c3f8  [heap]
I/DEBUG   (  937):     47c97b30  00001400  
I/DEBUG   (  937):     47c97b34  00221918  [heap]
I/DEBUG   (  937):     47c97b38  df002777  
I/DEBUG   (  937):     47c97b3c  e3a070ad  
I/DEBUG   (  937): #00 47c97b40  44384520  /dev/ashmem/mspace/dalvik-heap/2 (deleted)
I/DEBUG   (  937):     47c97b44  00205048  [heap]
I/DEBUG   (  937):     47c97b48  42944f8c  
I/DEBUG   (  937):     47c97b4c  80816df8  /system/lib/libdvm.so
I/DEBUG   (  937):     47c97b50  00221918  [heap]
I/DEBUG   (  937):     47c97b54  4125edb8  /dev/ashmem/dalvik-LinearAlloc (deleted)
I/DEBUG   (  937):     47c97b58  44384520  /dev/ashmem/mspace/dalvik-heap/2 (deleted)
I/DEBUG   (  937):     47c97b5c  42944f84  
I/DEBUG   (  937):     47c97b60  80816fc0  /system/lib/libdvm.so
I/DEBUG   (  937):     47c97b64  00221918  [heap]
I/DEBUG   (  937):     47c97b68  47c97b70  
I/DEBUG   (  937):     47c97b6c  80844977  /system/lib/libdvm.so
I/DEBUG   (  937):     47c97b70  42944f84  
I/DEBUG   (  937):     47c97b74  42936348  /data/dalvik-cache/data@[e-mail]@classes.dex
I/DEBUG   (  937):     47c97b78  80c19361  /data/data/org.mozilla.fennec/lib/libmozutils.so
I/DEBUG   (  937):     47c97b7c  47c97c28  
I/DEBUG   (  937):     47c97b80  47c97c28  
I/DEBUG   (  937):     47c97b84  00000001  
I/WindowManager( 1021): WIN DEATH: Window{444a0a48 org.mozilla.fennec/org.mozilla.fennec.App paused=false}
I/ActivityManager( 1021): Process org.mozilla.fennec (pid 4399) has died.
I/WindowManager( 1021): WIN DEATH: Window{44468b18 SurfaceView paused=false}

Comment 1

[Matt Brubeck (:mbrubeck)]

There was a fix for an Android debug startup crash checked in yesterday (bug 690856), but it looks like builds are still crashing.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

:blassey had mentioned a fix for debug builds and I waited until that fix produced a debug android build from m-c and tested it.

Comment 3

[Mike Hommey [:glandium]]

Which build was that exactly?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

# cat /data/data/org.mozilla.fennec/application.ini
[App]
Vendor=Mozilla
Name=Fennec
Version=10.0a1
BuildID=20110930120933
SourceRepository=http://hg.mozilla.org/mozilla-central
SourceStamp=1463dc6308a8
ID={a23983c0-fd0e-11dc-95ff-0800200c9a66}

[Gecko]
MinVersion=1.9.2b5pre
MaxVersion=10.0a1

[XRE]
EnableExtensionManager=1

[Crash Reporter]
Enabled=1
ServerURL=https://crash-reports.mozilla.com/submit
#

Comment 5

[Mike Hommey [:glandium]]

This is a call on a null pointer, which happens in the Java_org_mozilla_gecko_GeckoAppShell_nativeRun wrapper function in libmozutils.so, because it couldn't get the corresponding function from libxul.so. It can't do so because libxul.so wasn't loaded, because a bunch of libs weren't loaded.

All in all, figure out why you get this:
E/GeckoLibLoad( 4399): Couldn't load /data/data/org.mozilla.fennec/cache/libnss3.so because Cannot load library: link_image[2046]: failed to link libnss3.so

and you'll have it figured. I'll take a look on monday.

Comment 6

[Mike Hommey [:glandium]]

Interestingly, on my Asus Transformer, the same build doesn't crash in the main process, but the content process crashes in a similar way, but with a slightly different root cause:

10-02 00:04:26.940 E/GeckoLibLoad( 3019): Couldn't load /data/data/org.mozilla.fennec/cache/libnss3.so because Cannot load library: reloc_library[1315]:  3019 cannot locate '__aeabi_llsl'...

Comment 7

[Mike Hommey [:glandium]]

I can reliably reproduce with that build, but I can't reproduce on latest tinderbox debug build nor on a custom debug build (neither current trunc nor revision 1463dc6308a8) :(

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

ok, I tried a build from this morning and it seems to be working fine.  I assume we can close this bug?

Comment 9

[Mike Hommey [:glandium]]

No :( I just tried a m-i build (704f37801611) and it crashes too.

Comment 10

[Mike Hommey [:glandium]]

Attachment: dltest

On my Nexus S, depending on the build used, putting the attached dltest file in /data/local, the following command works or fails:
$ LD_LIBRARY_PATH=/data/data/org.mozilla.fennec/lib:/data/data/org.mozilla.fennec/cache /data/local/dltest /data/data/org.mozilla.fennec/cache/libnss3.so

When it fails, I get the following message:
Cannot load library: reloc_library[1311]:  <pid> cannot locate '__aeabi_llsl'...

All libnss3.so I tested (working and failing) do have an undefined __aeabi_llsl symbol, and all have a relocation for it. In theory, if the symbol could really not be located, we should see the error on all debug builds, but we don't.

Comment 11

[Mike Hommey [:glandium]]

So, here is what probably happens (you've got to love the toolchain):
- depending on optimization, other-licenses/android/linker_format.o requires __aeabi_llsl.
- when linking libmozutils.so, the linker pulls __aeabi_llsl from libgcc.a, and includes it.
- when linking libnss3.so, as some of its objects use __aeabi_llsl, the linker searches it in the various libraries, and finds it in libmozutils.so. It thus marks it undefined, generates a PLT for that function, and a corresponding GOT entry.
- now, something changes that makes other-licenses/android be rebuilt, and not include __aeabi_llsl.
- but libnss3.so is not rebuilt, so __aeabi_llsl is still undefined.
- at runtime, the linker goes through all relocations of libnss3.so, and can't find __aeabi_llsl in any of the libraries already loaded, since it's not in libmozutils.so anymore.

Workaround: clobber and rebuild.

Notes:
- clobber builds are unaffected. This includes nightlies. But we don't have debug nightly builds.
- on opt builds, we use our custom linker to load our libs instead of the system linker. Our linker does one thing that the system linker doesn't: it does dynamic resolution of symbols. IOW, we only resolve (some) symbols when they are needed. In practice, for this problem, this makes possible problem happen when __aeabi_llsl happens to be needed at run time, which, fortunately, seems to be not that often. This is why, by modifying a failing build, with an hex editor, so that it doesn't extract libraries and thus use our linker, I could get a working fennec. It would probably have crashed kind of randomly, much later.
- I had another kind of related issue on Debian powerpc, with a symbol that comes from libgcc.a and was defined by another library at link time. I think it's utterly stupid to apply common linking rules for things that are pretty much internal to the toolchain. I think the best way to avoid this would be to avoid __aeabi_* symbols to be exported from libmozutils.so.

Comment 12

[Brad Lassey [:blassey] (use needinfo?)]

glandium can you take a look at this?

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

it seems that doing a clobber build resolves this.