Closed Bug 1312545 Opened 8 years ago Closed 7 years ago

Pre-installed fennec fails to load libnss3.so

Categories

(Firefox for Android Graveyard :: General, defect)

51 Branch
defect
Not set
critical

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: fabrice, Unassigned)

References

Details

STR:
- prepare an AOSP build with fennec pre-installed.
- flash the device.
- launch fennec.

Observed:
- Up to gecko 50, everything works fine.
- Starting with 51, fennec crashes during startup with the following error:

09-06 00:41:17.016  2192  2192 D QSEECOMD: : qseecom listener services process entry PPID = 1
09-06 00:41:17.016  2192  2192 E QSEECOMD: : Listener: index = 0, hierarchy = 0
09-06 00:41:17.016  2192  2192 E QSEECOMD: : Init dlopen(librpmb.so, RLTD_NOW) is failed....
09-06 00:41:17.016  2192  2192 E QSEECOMD: : ERROR: RPMB_INIT failed, shall not start listener services
09-06 00:41:21.646  2194  2194 D AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 0 <<<<<<
09-06 00:41:21.650  2194  2194 D AndroidRuntime: CheckJNI is OFF
09-06 00:41:21.759  2194  2194 D ICU     : No timezone override file found: /data/misc/zoneinfo/current/icu/icu_tzdata.dat
09-06 00:41:21.797  2194  2194 I Radio-JNI: register_android_hardware_Radio DONE
09-06 00:41:21.817  2194  2194 D AndroidRuntime: Calling main entry com.android.commands.am.Am
09-06 00:41:21.819   778  1273 I ActivityManager: START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] flg=0x10000000 pkg=org.mozilla.fennec_aurora cmp=org.mozilla.fennec_aurora/.App} from uid 0 on display 0
09-06 00:41:21.833   778  1273 I ActivityManager: Start proc 2202:org.mozilla.fennec_aurora/u0a12 for activity org.mozilla.fennec_aurora/.App
09-06 00:41:21.836  2194  2194 D AndroidRuntime: Shutting down VM
09-06 00:41:21.868  2202  2202 W System  : ClassLoader referenced unknown path: /system/app/B2G/lib/arm
09-06 00:41:21.881  2202  2202 I GeckoApplication: zerdatime 323772 - Fennec application start
09-06 00:41:21.889   778  1270 I ActivityManager: START u0 {act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] pkg=org.mozilla.fennec_aurora cmp=org.mozilla.fennec_aurora/org.mozilla.gecko.BrowserApp} from uid 10012 on display 0
09-06 00:41:21.915  2202  2215 D GeckoSharedPrefs: Current version = 2, prefs version = 2
09-06 00:41:21.931  2202  2202 D GeckoScreenOrientation: updating to new orientation PORTRAIT_PRIMARY
09-06 00:41:21.969  2202  2217 D GeckoLoader: Gecko environment env0: null
05-12 01:17:20.421  2217  2217 W Gecko   : type=1400 audit(0.0:30): avc: denied { execute } for path="/data/data/org.mozilla.fennec_aurora/cache/libnss3.so" dev="mmcblk0p28" ino=81590 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=file permissive=0
09-06 00:41:21.975  2202  2217 E GeckoLibLoad: Load sqlite start
09-06 00:41:21.975  2202  2217 E GeckoLinker: /system/app/B2G/B2G.apk!/assets/armeabi-v7a/libnss3.so: Failed to mmap
09-06 00:41:21.976  2202  2217 E GeckoLibLoad: Couldn't get a handle to libnss3!
09-06 00:41:21.976  2202  2217 E GeckoLibLoad: Throw
09-06 00:41:21.976  2202  2217 E GeckoLibLoad: Load sqlite done
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler: >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 97 ("Gecko")
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler: java.lang.Exception: Error loading sqlite libraries
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibsNative(Native Method)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibs(GeckoLoader.java:222)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:398)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:496)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler: Main thread (1) stack:
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler:     java.lang.VMClassLoader.findLoadedClass(Native Method)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler:     java.lang.ClassLoader.findLoadedClass(ClassLoader.java:362)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler:     java.lang.ClassLoader.loadClass(ClassLoader.java:499)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler:     java.lang.ClassLoader.loadClass(ClassLoader.java:469)
09-06 00:41:21.976  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.createView(LayoutInflater.java:583)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:764)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:704)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:835)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:798)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:838)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:798)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:515)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:423)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:374)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.support.v7.app.AppCompatDelegateImplV7.setContentView(Unknown Source)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.support.v7.app.AppCompatActivity.setContentView(Unknown Source)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     org.mozilla.gecko.GeckoApp.onCreate(GeckoApp.java:1253)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     org.mozilla.gecko.BrowserApp.onCreate(BrowserApp.java:611)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.app.Activity.performCreate(Activity.java:6251)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1107)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2369)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2476)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.app.ActivityThread.-wrap11(ActivityThread.java)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.app.ActivityThread$H.handleMessage(ActivityThread.java:1344)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.os.Handler.dispatchMessage(Handler.java:102)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.os.Looper.loop(Looper.java:148)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     android.app.ActivityThread.main(ActivityThread.java:5417)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     java.lang.reflect.Method.invoke(Native Method)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
09-06 00:41:21.977  2202  2217 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)
09-06 00:41:21.977  2202  2217 E art     : No implementation found for void org.mozilla.gecko.GeckoAppShell.reportJavaCrash(java.lang.Throwable, java.lang.String) (tried Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash and Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash__Ljava_lang_Throwable_2Ljava_lang_String_2)
09-06 00:41:21.982   778  1273 I ActivityManager: START u0 {act=org.mozilla.gecko.reportCrash flg=0x10000000 cmp=org.mozilla.fennec_aurora/org.mozilla.gecko.CrashReporter (has extras)} from uid 10012 on display 0
09-06 00:41:21.983  2202  2217 I Process : Sending signal. PID: 2202 SIG: 9
09-06 00:41:21.999   778  1476 I ActivityManager: Process org.mozilla.fennec_aurora (pid 2202) has died


Some additional information:
- Fennec is installed under /system/app/fennec. There is a single fennec.apk file in this directory.
- /data/data/org.mozilla.fennec_aurora contains the following:
drwxrwx--x u0_a12   u0_a12            1970-09-06 00:40 app_plugins
drwxrwx--x u0_a12   u0_a12            1970-09-06 00:40 app_plugins_private
drwxrwx--x u0_a12   u0_a12            1970-09-06 00:40 app_tmpdir
drwxrwxrwx u0_a12   u0_a12            1970-09-06 00:41 cache
drwxrwx--x u0_a12   u0_a12            1970-09-06 00:40 code_cache
drwxrwx--x u0_a12   u0_a12            1970-09-06 00:37 files
lrwxrwxrwx root     root              1970-09-06 00:36 lib -> /system/app/fennec/lib/arm
drwxrwx--x u0_a12   u0_a12            1970-09-06 00:37 shared_prefs

It's quite clear the the lib symlink points to a non-existent location.
libnss3 can be found in the cache directory:

root@hammerhead:/data/data/org.mozilla.fennec_aurora # ls -l cache
-rw------- u0_a12   u0_a12          0 1970-09-06 00:41 ed795426-a3ac-47f8-94c9-bab91a292dc9.dmp
-rw------- u0_a12   u0_a12        708 1970-09-06 00:41 ed795426-a3ac-47f8-94c9-bab91a292dc9.extra
-rw------- u0_a12   u0_a12    1402280 1970-09-06 00:40 libnss3.so
-rw------- u0_a12   u0_a12          4 1970-09-06 00:40 libnss3.so.crc

needinfo on snorp & jim, in case they know which elf loader changes could have caused that. If not, I'll bisect.
Flags: needinfo?(snorp)
Flags: needinfo?(nchen)
A couple bugs were mentioned on IRC that might have affected this:

https://bugzil.la/1291424 — FIXED, esawin@mozilla.com — Consider removing on-demand decompression linker
https://bugzil.la/1298090 — FIXED, esawin@mozilla.com — Extract shared libraries on APK update

I'm marking this critical because it affects preload distributions.
Severity: normal → critical
Looks like esawin is actually the one I should ni?
Flags: needinfo?(esawin)
Yeah, those are the likely bugs. I see one interesting thing:

05-12 01:17:20.421  2217  2217 W Gecko   : type=1400 audit(0.0:30): avc: denied { execute } for path="/data/data/org.mozilla.fennec_aurora/cache/libnss3.so" dev="mmcblk0p28" ino=81590 scontext=u:r:platform_app:s0:c512,c768 tcontext=u:object_r:app_data_file:s0:c512,c768 tclass=file permissive=0

It looks like selinux is preventing us from mapping the library as executable. Before, we would've been doing this with anonymous memory and not a specific file. I don't see this message when launching Fennec with a normal install, so I think this may be where things are going south. Fabrice, maybe see about tweaking the selinux config?

09-06 00:41:21.975  2202  2217 E GeckoLinker: /system/app/B2G/B2G.apk!/assets/armeabi-v7a/libnss3.so: Failed to mmap

The path there is wrong, but I think that's just a busted error message? The message above indicates that we probably tried to open the right path.
Fabrice, I'm looking at the platform_app policy here[0] and it seems pretty restrictive compared to what I think is the 'normal' app policy at [1]. Maybe you can try launching Fennec differently? How are you starting it now?

[0] http://androidxref.com/7.0.0_r1/xref/system/sepolicy/platform_app.te
[1] http://androidxref.com/7.0.0_r1/xref/system/sepolicy/app.te
Flags: needinfo?(snorp) → needinfo?(fabrice)
I'm starting it with |am start org.mozilla.xxx|

You're right about the selinux configuration. I tested by changing the LOCAL_CERTIFICATE in the build config to "shared" instead of "platform" and that let us start. Obviously I'll have to check what the consequences are, but for my own experiment it's good enough!
Flags: needinfo?(fabrice)
The relevant change happened in bug 1291424.

> 09-06 00:41:21.868  2202  2202 W System  : ClassLoader referenced unknown
> path: /system/app/B2G/lib/arm
> 09-06 00:41:21.975  2202  2217 E GeckoLinker:
> /system/app/B2G/B2G.apk!/assets/armeabi-v7a/libnss3.so: Failed to mmap

These logs are worrying. Does changing the selinux config fix the package path?
The package path is passed down to the linker from here: https://dxr.mozilla.org/mozilla-central/source/mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoThread.java#395
Blocks: 1291424
Flags: needinfo?(esawin)
Flags: needinfo?(nchen)
Using a releasekey fixes the issue too. I would be surprised if partners needed anything else, so if no one objects I think we can close this bug.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INVALID
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.