Startup crash in java.lang.Exception: Error loading gecko libraries at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)

RESOLVED FIXED in Firefox 51

Status

()

defect
--
critical
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: ashughes, Assigned: esawin)

Tracking

({crash, topcrash-android-armv7})

unspecified
Firefox 53
All
Android
Points:
---

Firefox Tracking Flags

(firefox47- wontfix, firefox48+ wontfix, firefox49+ wontfix, fennec+, firefox50+ wontfix, firefox51+ fixed, firefox52+ fixed, firefox53 fixed)

Details

(crash signature)

Attachments

(1 attachment, 1 obsolete attachment)

This bug was filed from the Socorro interface and is 
report bp-83802b95-bc7c-4ec2-ba16-ab2142160625.
=============================================================
java.lang.Exception: Error loading gecko libraries
	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)
	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibs(GeckoLoader.java:509)
	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:331)
	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:426)
=============================================================
More reports: https://crash-stats.mozilla.com/signature/?product=FennecAndroid&signature=java.lang.Exception%3A%20Error%20loading%20gecko%20libraries%20at%20org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative%28Native%20Method%29

Looks like bug 1089260 has come back recently. This is now the #1 crash in Fennec 47, accounting for 7% of the overall volume with over 41K reports in the last week and is a startup crash as well. It looks like there is a strong correlation to Sony and LG devices, with the Sony D5803 as the top device (12%).
[Tracking Requested - why for this release]: #1 Fennec topcrash and a startup crash. Likely impacts user retention.
The only relevant changes to the Android linker code in 47 have been in bug 1239789, might be a regression?
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(droeh)
There is not much to go by in the crash reports... can someone reproduce and get a more complete log?
Flags: needinfo?(mh+mozilla)
Looks like it affects mostly Sony Z3-Z5 (Compact) devices, I don't have any of those to reproduce.
I took a look at my patch for bug 1239789 and I'm not sure how it could cause this, but I agree that it looks like the most relevant change in 47... this may be difficult to approach without being able to reproduce.
Flags: needinfo?(droeh)
tracking-fennec: --- → ?
I found one report with a logcat:

07-07 18:30:40.539 21422 21444 E GeckoLibLoad: Load sqlite done
07-07 18:30:40.540 21422 21444 E GeckoLibLoad: Load nss start
07-07 18:30:40.540 21422 21444 E GeckoLibLoad: Load nss done
07-07 18:30:40.549 21422 21444 E GeckoLinker: Malformed or broken seekable zstream
07-07 18:30:40.551 21422 21444 E GeckoLibLoad: Couldn't get a handle to libxul!
07-07 18:30:40.551 21422 21444 E GeckoLibLoad: Throw
07-07 18:30:40.552 21422 21444 E GeckoCrashHandler: >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 24328 ("Gecko")
07-07 18:30:40.552 21422 21444 E GeckoCrashHandler: java.lang.Exception: Error loading gecko libraries
07-07 18:30:40.552 21422 21444 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)
07-07 18:30:40.552 21422 21444 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibs(GeckoLoader.java:509)
07-07 18:30:40.552 21422 21444 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:331)
07-07 18:30:40.552 21422 21444 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:426)
07-07 18:30:40.552 21422 21444 E GeckoCrashHandler: Main thread (1) stack:
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     org.mozilla.gecko.permissions.Permissions.<clinit>(Permissions.java:46)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     org.mozilla.gecko.BrowserApp.onResume(BrowserApp.java:1039)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1257)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.Activity.performResume(Activity.java:6076)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.ActivityThread.performResumeActivity(ActivityThread.java:3039)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.ActivityThread.handleResumeActivity(ActivityThread.java:3081)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2447)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.ActivityThread.access$800(ActivityThread.java:156)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.ActivityThread$H.handleMessage(ActivityThread.java:1351)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.os.Handler.dispatchMessage(Handler.java:102)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.os.Looper.loop(Looper.java:211)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     android.app.ActivityThread.main(ActivityThread.java:5389)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     java.lang.reflect.Method.invoke(Native Method)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     java.lang.reflect.Method.invoke(Method.java:372)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1020)
07-07 18:30:40.562 21422 21444 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:815)
Here's another one:

07-07 08:16:41.506 27532 27556 E GeckoLibLoad: Load sqlite start
07-07 08:16:41.528 27532 27551 D GeckoSearchEngineManager: Found default engine name in SharedPreferences: Google
07-07 08:16:41.530 27532 27556 W GeckoLinker: /data/app/org.mozilla.firefox_beta-1/base.apk!/assets/armeabi-v7a/libnss3.so: unhandled flags #8 not handled
07-07 08:16:41.540 27532 27556 W GeckoLinker: /data/app/org.mozilla.firefox_beta-1/base.apk!/assets/armeabi-v7a/libnss3.so: Relocation to NULL @0x00147fe0 for symbol "__cxa_type_match"
07-07 08:16:41.544 27532 27556 E GeckoLibLoad: Load sqlite done
07-07 08:16:41.544 27532 27556 E GeckoLibLoad: Load nss start
07-07 08:16:41.544 27532 27556 E GeckoLibLoad: Load nss done
07-07 08:16:41.550 27532 27556 E GeckoLinker: inflate failed: invalid distance too far back
07-07 08:16:41.551 27532 27556 E GeckoLibLoad: Couldn't get a handle to libxul!
07-07 08:16:41.551 27532 27556 E GeckoLibLoad: Throw
07-07 08:16:41.551 27532 27556 E GeckoCrashHandler: >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 4339 ("Gecko")
07-07 08:16:41.551 27532 27556 E GeckoCrashHandler: java.lang.Exception: Error loading gecko libraries
07-07 08:16:41.551 27532 27556 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)
07-07 08:16:41.551 27532 27556 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibs(GeckoLoader.java:509)
07-07 08:16:41.551 27532 27556 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:370)
07-07 08:16:41.551 27532 27556 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:491)
07-07 08:16:41.551 27532 27556 E GeckoCrashHandler: Main thread (1) stack:
07-07 08:16:41.563 27532 27551 D GeckoSessInfo: Recording start of session: 1467875801536
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.media.MediaRouter.<init>(MediaRouter.java:719)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.app.SystemServiceRegistry$7.createService(SystemServiceRegistry.java:193)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.app.SystemServiceRegistry$7.createService(SystemServiceRegistry.java:192)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.app.SystemServiceRegistry$CachedServiceFetcher.getService(SystemServiceRegistry.java:779)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.app.SystemServiceRegistry.getSystemService(SystemServiceRegistry.java:732)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.app.ContextImpl.getSystemService(ContextImpl.java:1385)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.content.ContextWrapper.getSystemService(ContextWrapper.java:633)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v7.media.MediaRouterJellybean.getMediaRouter(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v7.media.SystemMediaRouteProvider$JellybeanImpl.<init>(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v7.media.SystemMediaRouteProvider$JellybeanMr1Impl.<init>(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v7.media.SystemMediaRouteProvider$JellybeanMr2Impl.<init>(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v7.media.SystemMediaRouteProvider.obtain(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v7.media.MediaRouter$GlobalMediaRouter.<init>(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v7.media.MediaRouter.getInstance(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     org.mozilla.gecko.MediaPlayerManager.onResume(MediaPlayerManager.java:224)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v4.app.Fragment.performResume(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v4.app.FragmentManagerImpl.moveToState(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v4.app.FragmentManagerImpl.moveToState(Unknown Source)
07-07 08:16:41.565 27532 27556 E GeckoCrashHandler:     android.support.v4.app.FragmentManagerImpl.moveToState$2563266(Unknown Source)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.support.v4.app.FragmentManagerImpl.dispatchResume(Unknown Source)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.support.v4.app.FragmentActivity.onResumeFragments(Unknown Source)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.support.v4.app.FragmentActivity.onPostResume(Unknown Source)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.support.v7.app.AppCompatActivity.onPostResume(Unknown Source)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.app.Activity.performResume(Activity.java:6351)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.app.ActivityThread.performResumeActivity(ActivityThread.java:3120)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.app.ActivityThread.handleResumeActivity(ActivityThread.java:3162)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2509)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.app.ActivityThread.-wrap11(ActivityThread.java)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.app.ActivityThread$H.handleMessage(ActivityThread.java:1347)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.os.Handler.dispatchMessage(Handler.java:102)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.os.Looper.loop(Looper.java:148)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     android.app.ActivityThread.main(ActivityThread.java:5458)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     java.lang.reflect.Method.invoke(Native Method)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
07-07 08:16:41.566 27532 27556 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)
:glandium, do you have any ideas?
Flags: needinfo?(mh+mozilla)
> 07-07 18:30:40.549 21422 21444 E GeckoLinker: Malformed or broken seekable zstream

This error message comes from https://dxr.mozilla.org/mozilla-central/rev/214884d507ee369c1cf14edb26527c4f9a97bf48/mozglue/linker/SeekableZStream.cpp#36

> 07-07 08:16:41.550 27532 27556 E GeckoLinker: inflate failed: invalid distance too far back

This one comes from https://dxr.mozilla.org/mozilla-central/rev/214884d507ee369c1cf14edb26527c4f9a97bf48/mozglue/linker/SeekableZStream.cpp#94

The source data there is mmap()ed, I don't see how that could be corrupted other than the APK being corrupted :-/ RAM chip problems maybe?
Flags: needinfo?(mh+mozilla)
I hit this once today on 48.0b6 20160707040648. My crash report https://crash-stats.mozilla.com/report/index/595f2dc2-b310-4cb4-b7c0-9e3682160713
If you ever can reproduce, it would be useful if you ran fennec with MOZ_DEBUG_LINKER set to 1 (however that can be switched on nowadays, -> snorp? (I don't think we have a switch that turn it on for every time fennec is started, sadly)), and attached the logcat from around the crash when you reproduce it.
Flags: needinfo?(snorp)
Looks like 48 and 49 may also be affected.  Tracking since this crash showed up so strongly on the release channel. 
At the channel meeting Mihai mentioned that he and his team are going to investigate this a bit further.
According to crash stats 50 is affected, tracking 50+ for this startup crash.
Still the #1 topcrash on Fennec.
This seems bad, but I don't see anything actionable. Maybe some bad interaction with play store updating?
tracking-fennec: ? → +
Flags: needinfo?(snorp)
Hi Florin,
Can we have QA to help to reproduce the issue based on comment #11?
Flags: needinfo?(florin.mezei)
Forwarding to Ioana who leads the Mobile QA team. Ioana, is there something we can do here to try and reproduce?
Flags: needinfo?(florin.mezei) → needinfo?(ioana.chiorean)
http://mashable.com/ which redirects me to mashable.com India causes this crash for me   
I had an additional empty tab open and was just scrolling the mashable.com page up and down along with closing and opening the hamburger menu

https://crash-stats.mozilla.com/report/index/ac68efe9-4bcb-4447-83d6-8aea72160709

Another https://crash-stats.mozilla.com/report/index/77da48a3-f3c4-4014-b6c2-454272160712
I am not able to reproduce it on some of the devices we have in house ( even if they are on the device list in crashes) 
From comments I got two behaviors:
- crash when opening external link 
- crash while watching a (youtube) video

Tried to play a lot with mashable as in comment #19 but even if it is a heavy page it did not crash for me at all.
Flags: needinfo?(ioana.chiorean)
This is still a big startup crash in 48. It is only ~100 crashes in beta 6, but that puts it at the #4 topcrash in fennec beta. So this may still be a bad crash when we release 49.
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> This is still a big startup crash in 48. It is only ~100 crashes in beta 6,
> but that puts it at the #4 topcrash in fennec beta. So this may still be a
> bad crash when we release 49.

If you look at the devices tab in the summary, you will also see this affects a very wide spectrum of phones and tablets.

Some of the comments seem to point to actions involving clicking a link:

clicked a link in gmail, Firefox crashed 
followed a facebook link that opened it Firefox which crashed right away. 
I was trying to open an article from my Google news. 
Wasn't doing anything in particular. Was just trying to open a page from a Google search (from Google app on Android, launching Firefox). 
just opening via news feed 
crashed after being on standby all night huh
(In reply to Marcia Knous [:marcia - use ni] from comment #22)
> (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> > This is still a big startup crash in 48. It is only ~100 crashes in beta 6,
> > but that puts it at the #4 topcrash in fennec beta. So this may still be a
> > bad crash when we release 49.
> 
> If you look at the devices tab in the summary, you will also see this
> affects a very wide spectrum of phones and tablets.
> 
> Some of the comments seem to point to actions involving clicking a link:
> 
> clicked a link in gmail, Firefox crashed 
> followed a facebook link that opened it Firefox which crashed right away. 
> I was trying to open an article from my Google news. 
> Wasn't doing anything in particular. Was just trying to open a page from a
> Google search (from Google app on Android, launching Firefox). 
> just opening via news feed 
> crashed after being on standby all night huh

It's not surprising though, because click a link in other apps usually need to restart firefox. At least on my z3c firefox is easily OOM killed by foreground apps.
Sounds like we don't have any traction here yet.  Wontfix for 49. 

Sebastian can you think of any other way to investigate this? I'm not blocking 49 release on it though it is our top crash - in part because the issue seems to have been around for a while and we have wontfixed it at least twice.
Flags: needinfo?(s.kaspari)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #24)
> Sebastian can you think of any other way to investigate this? I'm not
> blocking 49 release on it though it is our top crash - in part because the
> issue seems to have been around for a while and we have wontfixed it at
> least twice.

Unfortunately I do not have any more ideas.

It's interesting that the majority of the crashes (12.5%) are from the Sony Z3C (D5803) running API 23 (Android 6). I've a Z3C here but never got an OTA for 6.0 (It's still running Android 5.1). It seems like images are available from Sony though; I'll perform a manual update and see what happens.
Flags: needinfo?(s.kaspari)
Any progress, Sebastian? (I know you just got back from PTO so no hurry; just making sure it's still on your radar)
Flags: needinfo?(s.kaspari)
I think it's likely bug 1302516 could cause this type of crash. We should uplift that and see if this crash improves.
(In reply to Jim Chen [:jchen] [:darchons] from comment #26)
> Any progress, Sebastian? (I know you just got back from PTO so no hurry;
> just making sure it's still on your radar)

Yeah, unfortunately I wasn't able to upgrade using Sony's official tools because the device was already rooted (it has been running Firefox OS previously). Somehow I'd need to get it back into stock state and this seemed to be more complicated.
Flags: needinfo?(s.kaspari)
According to this crash report: https://crash-stats.mozilla.com/report/index/dde6f0da-f8bc-4bd9-83eb-cdf7c2161006
fennec aurora is still affected (the build is from yesterday, so, after bug 1302516 landed).

Snorp, any new idea?
Flags: needinfo?(snorp)
Bummer. We could try uplifting bug 1291424, which changes how we load the libraries entirely.
Flags: needinfo?(snorp)
I'm still seeing this crash on Nightly, so bug 1291424 hasn't fixed it either.
This crash has been there for several releases, a quick fix seems unlikely, and 50 is approaching RC stage, tagging wontfix. :(
This bug covers a broad spectrum of possible issues since most linker failures will create this signature. We probably should find a way to refine this by creating more specific crash signatures.

With bug 1294731, we've introduced checksum-based cache file validation. Should Android evict a cached library file but leave its .crc file in cache, we would fail to re-create the library file and crash. This could be one of the new issues contributing to this signature.
Assignee: nobody → esawin
Attachment #8809498 - Flags: review?(mh+mozilla)
Comment on attachment 8809498 [details] [diff] [review]
0001-Bug-1283629-1.0-Check-for-cached-library-file-availa.patch

Review of attachment 8809498 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozglue/linker/Mappable.cpp
@@ +63,5 @@
> +    if (memcmp(checksumBuf, &mChecksum, sizeof(mChecksum))) {
> +      return false;
> +    }
> +    AutoCloseFD fd;
> +    fd = open(aCachedLibPath, O_RDONLY | O_NOATIME, S_IRUSR | S_IWUSR);

stat() or access() would be better than opening the file. You also don't really need the file path as an argument, you can derive it from the checksum file name you have as a member variable.
Attachment #8809498 - Flags: review?(mh+mozilla)
Use access to check for file availability (and readability) and keep the cached library path as a member to avoid additional string acrobatics.
Attachment #8809498 - Attachment is obsolete: true
Attachment #8809654 - Flags: review?(mh+mozilla)
OS: Unspecified → Android
Hardware: Unspecified → All
Attachment #8809654 - Flags: review?(mh+mozilla) → review+
Pushed by esawin@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b7d42c3980d1
[1.1] Check for cached library file availability before reusing it. r=glandium
https://hg.mozilla.org/mozilla-central/rev/b7d42c3980d1
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 53
Track 51+/52+ as topcrash.
Hi :esawin,
Since this also affects 51 and 52, do you think the patch is worth uplifting to 51 & 52?
Flags: needinfo?(esawin)
The fix won't have a dramatic effect on the crash rate with this signature, but it's safe to uplift.
Flags: needinfo?(esawin)
Comment on attachment 8809654 [details] [diff] [review]
0001-Bug-1283629-1.1-Check-for-cached-library-file-availa.patch

Approval Request Comment
[Feature/regressing bug #]: Bug 1294731.
[User impact if declined]: Potential crash on startup.
[Describe test coverage new/current, TreeHerder]: Nightly.
[Risks and why]: Minimal, we're only checking for a file's existence.
[String/UUID change made/needed]: None.
Attachment #8809654 - Flags: approval-mozilla-beta?
Attachment #8809654 - Flags: approval-mozilla-aurora?
Comment on attachment 8809654 [details] [diff] [review]
0001-Bug-1283629-1.1-Check-for-cached-library-file-availa.patch

Take this patch and see how it improves. Beta51+ and Aurora52+. Should be in 51 beta 2.
Attachment #8809654 - Flags: approval-mozilla-beta?
Attachment #8809654 - Flags: approval-mozilla-beta+
Attachment #8809654 - Flags: approval-mozilla-aurora?
Attachment #8809654 - Flags: approval-mozilla-aurora+
See Also: → 1337290
You need to log in before you can comment on or make changes to this bug.