1042935 - Unable to load mozglue

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect, P5)

Description

[Richard Newman [:rnewman]]

This is a follow-up from Bug 990130.

We're seeing lots of crashes like the one below.

This means that our smart library loader first tries

              System.loadLibrary(lib);

then tries

                System.load("/data/app-lib/" + ANDROID_PACKAGE_NAME + "/lib" + lib + ".so");

then tries

                    System.load("/data/data/" + ANDROID_PACKAGE_NAME + "/lib/lib" + lib + ".so");

When that fails, it throws the "Unable to load" error.


the loadLibrary call shouldn't fail in the first place -- the behavior here is congruent with libmozglue being completely absent from the device.



java.lang.RuntimeException: Unable to create application org.mozilla.gecko.GeckoApplication: java.lang.RuntimeException: Unable to load mozglue
at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4558)
at android.app.ActivityThread.access$1300(ActivityThread.java:149)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1353)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:137)
at android.app.ActivityThread.main(ActivityThread.java:5214)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:525)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:739)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:555)
at dalvik.system.NativeStart.main(Native Method)
Caused by: java.lang.RuntimeException: Unable to load mozglue
at org.mozilla.gecko.mozglue.GeckoLoader.doLoadLibrary(GeckoLoader.java:270)
at org.mozilla.gecko.mozglue.GeckoLoader.loadMozGlue(GeckoLoader.java:284)
at org.mozilla.gecko.GeckoApplication.onCreate(GeckoApplication.java:123)
at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1007)
at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4555)
... 10 more
Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/data/org.mozilla.firefox/lib/libmozglue.so" not found
at java.lang.Runtime.load(Runtime.java:330)
at java.lang.System.load(System.java:511)
at org.mozilla.gecko.mozglue.GeckoLoader.doLoadLibrary(GeckoLoader.java:267)
... 14 more

Comment 1

[Brad Lassey [:blassey] (use needinfo?)]

one interesting thing to log would be if "/data/app-lib/" + ANDROID_PACKAGE_NAME + "/lib" + lib + ".so" or "/data/data/" + ANDROID_PACKAGE_NAME + "/lib/lib" + lib + ".so" exist as files. If they do and we can't load them then there is some library dependency that the system is missing. Those missing symbols should be listed in the log though.

There may be one last last ditch thing to do as well. We could manually extract the mozglue library from the jar and load it directly (this would be the fallback if neither of those files exist on disk).

Comment 2

[Sebastian Dröge (:slomo)]

Maybe related to https://stackoverflow.com/questions/18111739/why-do-some-android-phones-cause-our-app-to-throw-an-java-lang-unsatisfiedlinker aka https://code.google.com/p/android/issues/detail?id=35962 ?

Comment 3

[Richard Newman [:rnewman]]

Yes, as noted in Bug 990130 Comment 2, where we implemented the fallbacks mentioned in Comment 0.

Comment 4

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: crash-linkerror.png

Most common manifestation:
Caused by: java.lang.UnsatisfiedLinkError: Cannot load library: load_library(linker.cpp:745): library "/data/data/org.mozilla.firefox/lib/libmozglue.so" not found

Comment 5

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: crash-dlopen.png

Another common one:
Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: library "/data/data/org.mozilla.firefox/lib/libmozglue.so" not found

Comment 6

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: crash-issetguid.png

Can happen in significant numbers:
Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "issetugid" referenced by "libmozglue.so"...

Comment 7

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: crash-notvalidelf.png

Less common:
Caused by: java.lang.UnsatisfiedLinkError: Cannot load library: get_lib_extents[752]: 309 - /data/data/org.mozilla.firefox/lib/libmozglue.so is not a valid ELF object

Comment 8

[Mark Finkle (:mfinkle) (use needinfo?)]

Attachment: crash-dthash.png

Also rare:
Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: empty/missing DT_HASH in "libmozglue.so" (built with --hash-style=gnu?)

Comment 9

[Richard Newman [:rnewman]]

The issetugid is being tracked for L: Bug 1030899.

I also remember something similar to that in KitKat; maybe snorp remembers where my bugmail fails me?

Comment 10

[Richard Newman [:rnewman]]

(In reply to Mark Finkle (:mfinkle) from comment #5)

> Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: library
> "/data/data/org.mozilla.firefox/lib/libmozglue.so" not found

/data/data was our last fallback search path, so this just means "you couldn't load it for some other reason, and your last guess really sucks".

The amended version in Bug 1042984 throws the first exception, which will help.

Comment 11

[Richard Newman [:rnewman]]

> Caused by: java.lang.UnsatisfiedLinkError: dlopen failed: empty/missing
> DT_HASH in "libmozglue.so" (built with --hash-style=gnu?)

This should never never never happen -- we don't specify a hash-style, and the default in the NDK is correct for the Android dynamic linker -- but there are a few scattered reports for this online.

I would be inclined to think that this is a cloned/modified APK or a broken custom build (someone setting LDFLAGS?), except that these are both Galaxy S3s from US carriers.

Weird.

Comment 12

[Richard Newman [:rnewman]]

Data has started coming back from Beta.

LOAD mozglue: Data: /data/data/org.mozilla.firefox_beta, ax=false, ddx=true, -1x=false, -2x=false, nativeLib: /mnt/asec/org.mozilla.firefox_beta-1/lib, dirx=true, libx=true

The main library failed to load:

Caused by: java.lang.UnsatisfiedLinkError: Cannot load library: load_library(linker.cpp:784): not a valid ELF executable: /mnt/asec/org.mozilla.firefox_beta-1/lib/libmozglue.so

Comment 13

[Richard Newman [:rnewman]]

Wild and crazy searching:

https://code.google.com/p/webrtc/issues/detail?id=3332
https://code.google.com/p/android/issues/detail?id=59420

Comment 14

[Brad Lassey [:blassey] (use needinfo?)]

Can we add two things to this logging. First, log whether the APK is an x86 build or not and the arch of the device.

Second, can we compare the libmozglue.so on disk with the one in the APK. Maybe generate a checksum for both?

Comment 15

[Richard Newman [:rnewman]]

Some more:

LOAD mozglue: Data: /data/data/org.mozilla.firefox_beta, ax=false, ddx=false, -1x=false, -2x=false, nativeLib: /mismatched_uid/settings_10109/fs_10127, dirx=false, libx=false
Caused by: java.lang.UnsatisfiedLinkError: Couldn't load mozglue: findLibrary returned null

LOAD mozglue: Data: /data/data/org.mozilla.firefox_beta, ax=false, ddx=true, -1x=false, -2x=false, nativeLib: /mnt/asec/org.mozilla.firefox_beta-2/lib, dirx=true, libx=true
Cannot load library: load_library(linker.cpp:761): not a valid ELF executable: /mnt/asec/org.mozilla.firefox_beta-2/lib/libmozglue.so

LOAD mozglue: Data: /data/data/org.mozilla.firefox_beta, ax=false, ddx=false, -1x=false, -2x=false, nativeLib: /data/app-lib/org.mozilla.firefox_beta-1, dirx=true, libx=false
Couldn't load mozglue from loader dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/org.mozilla.firefox_beta-1.apk"],nativeLibraryDirectories=[/data/app-lib/org.mozilla.firefox_beta-1, /vendor/lib, /system/lib]]]: findLibrary returned null

Comment 16

[Richard Newman [:rnewman]]

The mismatched_uid one is interesting.

http://stackoverflow.com/questions/9037966/android-ndk-java-lang-unsatisfiedlinkerror-findlibrary-returned-null

"""
just had a similar problem.

Check out your /data/data/your.package.name/lib directory

When i ls in my package directory it currently displays:

lib -> /mismatched_uid/settings_10037/fs_1000

probably i accidently switched the sharedUserId and thus the library can't be accessed anymore.
"""


We do used shared user IDs. I wonder if this is related to the stuff we've seen with apps bizarrely trying to load a different Firefox's content providers?


It looks from other reading (e.g., <http://forum.swype.com/archive/index.php/t-4689.html>, <http://forum.xda-developers.com/showthread.php?t=1844868>) that perhaps this occurs when there's a collision of some kind during upgrade or restore, and Android helpfully assigns a new UID. Which of course doesn't work.

That's done in com/android/server/pm/PackageManagerService.java, when this conditional applies:

                // If we have mismatched owners for the data path, we have a problem.
                if (currentUid != pkg.applicationInfo.uid) {

so... kinda screwed.

Comment 17

[Richard Newman [:rnewman]]

nativeLib: /data/app-lib/org.mozilla.firefox_beta-1, dirx=true, libx=false

This looks like the classic case we'd be expecting to cause this error: the library file doesn't exist in the library search paths (which the PathClassLoader helpfully tells us are [/data/app-lib/org.mozilla.firefox_beta-1, /vendor/lib, /system/lib]).

It's possible that we can fix this (and maybe other occurrences) if we're allowed to write to that location and can copy files out of the APK ourselves.

Comment 18

[Brad Lassey [:blassey] (use needinfo?)]

filter on [mass-p5]

Comment 19

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.