1391268 - Startup crash in java.lang.Exception: Error loading gecko libraries at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, defect, P5)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-2ae3594f-4787-4a11-8e5a-ace0b0170817.
=============================================================

Seen while looking at release crash stats: http://bit.ly/2ia6IWC

Similar crash seen back in Firefox 53 and was fixed. Wide spectrum of devices are affected, ranging from API 24 to API 15.

ni on jchen since he addressed the last crash

Comments:

*It crashes numerous times especially when more than one window is in use.

Comment 1

[(inactive) Jim Chen [:jchen] [:darchons]]

I'll keep an eye on it.

Comment 2

[Intermittent Failures Robot]

1 failures in 908 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* android-7-1-armv8-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1391268&startday=2017-08-21&endday=2017-08-27&tree=all

Comment 3

[Marcia Knous [:marcia]]

Crash rate was high in 57.0.4 and is starting to grow as we have moved to 58. java.lang.Exception: Error loading gecko libraries at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method) leads 58 crash stats right now. Variety of devices, including blackberry devices (do we even support RIM?). Comments mention crashing when updating.

Jim - any ideas here? Thanks.

Comment 4

[:sdaswani]

Mike Comella took a look at this and from his analysis this is not a front-end problem. It seems to him that there the library loading is in front-end code but the cause is most often something back-end, e.g. they’re not compiled correctly, it’s using the wrong device architecture, etc., and it's best handled by the platform team.

Comment 5

[Marcia Knous [:marcia]]

Adding snorp per Comment 4, and Andreas as well.

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

I saw the following in a couple of the crashes that had logcats:

01-24 21:54:17.194  4613  4635 E GeckoLinker: XZ decoding: corrupt input stream
01-24 21:54:17.194  4613  4635 E GeckoLinker: Error decoding XZ file /data/user/0/org.mozilla.firefox/cache/libxul.so
01-24 21:54:17.300  4613  4635 E GeckoLibLoad: Couldn't get a handle to libxul!

Either the APK is corrupted or we still have bad races in the zip code. Eugen, do you still know of any issues there?

Comment 7

[James Willcox (:snorp) ([e-mail]) (he/him)]

It actually fails twice in these logs, because we clear the cache and try again. This seems to indicate a problem with the data on the disk.

I also found one user who encountered the same crash 5 times in a row yesterday. I think this backs up the disk corruption theory. The xz inflation seems to work for other libs on the same device (libnss, etc).

Comment 8

[James Willcox (:snorp) ([e-mail]) (he/him)]

Longer logcat snippet:

01-24 08:56:55.741 28770 28770 I GeckoApplication: zerdatime 2164487342 - application start
01-24 08:56:55.797 28770 28770 D GeckoSharedPrefs: Current version = 2, prefs version = 2
01-24 08:56:55.938 28770 28788 I GeckoThread: preparing to run Gecko
01-24 08:56:55.938 28770 28788 D GeckoLoader: Gecko environment env0: null
01-24 08:56:55.940 28770 28770 D GeckoScreenOrientation: updating to new orientation PORTRAIT_PRIMARY
01-24 08:56:55.946 28770 28788 E GeckoLibLoad: Load sqlite start
01-24 08:56:56.105 28770 28770 D Choreographer: init sf_choreo_doframe   debug_Level : 0x4f4cdebug_game_running : false
01-24 08:56:56.115 28770 28787 D GeckoProfile: Found profile dir.
01-24 08:56:56.117 28770 28787 E GeckoApp: An error occurred during restore, switching to backup file
01-24 08:56:56.117 28770 28787 E GeckoApp: org.mozilla.gecko.GeckoApp$SessionRestoreException: Could not read from session file
01-24 08:56:56.117 28770 28787 E GeckoApp: 	at org.mozilla.gecko.GeckoApp.restoreSessionTabs(GeckoApp.java:1722)
01-24 08:56:56.117 28770 28787 E GeckoApp: 	at org.mozilla.gecko.GeckoApp.access$400(GeckoApp.java:122)
01-24 08:56:56.117 28770 28787 E GeckoApp: 	at org.mozilla.gecko.GeckoApp$13.run(GeckoApp.java:1267)
01-24 08:56:56.117 28770 28787 E GeckoApp: 	at android.os.Handler.handleCallback(Handler.java:751)
01-24 08:56:56.117 28770 28787 E GeckoApp: 	at android.os.Handler.dispatchMessage(Handler.java:95)
01-24 08:56:56.117 28770 28787 E GeckoApp: 	at android.os.Looper.loop(Looper.java:154)
01-24 08:56:56.117 28770 28787 E GeckoApp: 	at org.mozilla.gecko.util.GeckoBackgroundThread.run(GeckoBackgroundThread.java:43)
01-24 08:56:56.122 28770 28788 W GeckoLinker: /data/app/org.mozilla.firefox-2/base.apk!/assets/armeabi-v7a/libnss3.so: unhandled flags #8 not handled
01-24 08:56:56.129 28770 28788 E GeckoLibLoad: Load sqlite done
01-24 08:56:56.129 28770 28788 E GeckoLibLoad: Load nss start
01-24 08:56:56.129 28770 28788 E GeckoLibLoad: Load nss done
01-24 08:56:56.162 28770 28790 D NetworkSecurityConfig: No Network Security Config specified, using platform default
01-24 08:56:56.172 28770 28787 D GeckoSessInfo: Building SessionInformation from prefs: 1516779755379, 0, true, false
01-24 08:56:56.172 28770 28790 I System.out: (HTTPLog)-Static: isSBSettingEnabled false
01-24 08:56:56.173 28770 28787 D GeckoApp: OS locale is de_DE, app locale is null
01-24 08:56:56.173 28770 28790 I System.out: (HTTPLog)-Static: isSBSettingEnabled false
01-24 08:56:56.173 28770 28787 D GeckoLocales: Previous locale de_DE same as new. Doing nothing.
01-24 08:56:56.184 28770 28770 D GeckoSysInfo: System memory: 1822MB.
01-24 08:56:56.197 28770 28770 F Adjust  : PRODUCTION: Adjust is running in Production mode. Use this setting only for the build that you want to publish. Set the environment to `sandbox` if you want to test your app!
01-24 08:56:56.221 28770 28770 D GeckoNetworkManager: Incoming event start for state OffNoListeners -> OnNoListeners
01-24 08:56:56.223 28770 28770 D GeckoNetworkManager: New network state: UP, WIFI, WIFI
--------- beginning of system
01-24 08:56:56.232 28770 28793 E ActivityThread: Failed to find provider info for com.facebook.katana.provider.AttributionIdProvider
01-24 08:56:56.245 28770 28787 D GeckoSessInfo: Recording start of session: 1516780616232
01-24 08:56:56.246 28770 28787 D GeckoSearchEngineManager: Found default engine name in SharedPreferences: Google
01-24 08:56:56.277 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: ThreadedRenderer.create() translucent=false
01-24 08:56:56.282 28770 28770 D InputTransport: Input channel constructed: fd=69
01-24 08:56:56.282 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: setView = DecorView@ab2fe1[BrowserApp] touchMode=true
01-24 08:56:56.306 28770 28797 D GeckoDLCCatalog: Loading from disk
01-24 08:56:56.306 28770 28796 D DLCStudyAction: Studying catalog..
01-24 08:56:56.314 28770 28770 D StumblerSafeReceiver: Registered local preference listener
01-24 08:56:56.325 28770 28797 D GeckoDLCCatalog: Loaded 48 elements
01-24 08:56:56.337 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: dispatchAttachedToWindow
01-24 08:56:56.356 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: Relayout returned: oldFrame=[0,0][0,0] newFrame=[0,0][1200,1920] result=0x7 surface={isValid=true -472647680} surfaceGenerationChanged=true
01-24 08:56:56.356 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: mHardwareRenderer.initialize() mSurface={isValid=true -472647680} hwInitialized=true
01-24 08:56:56.384 28770 28795 I OpenGLRenderer: Initialized EGL, version 1.4
01-24 08:56:56.384 28770 28795 D OpenGLRenderer: Swap behavior 1
01-24 08:56:56.393 28770 28795 D mali_winsys: EGLint new_window_surface(egl_winsys_display*, void*, EGLSurface, EGLConfig, egl_winsys_surface**, egl_color_buffer_format*, EGLBoolean) returns 0x3000,  [1200x1920]-format:1
01-24 08:56:56.421 28770 28770 D GeckoToolbar: onTabChanged: RESTORED
01-24 08:56:56.437 28770 28770 D SurfaceView: Relayout returned: oldFrame=[0,0][0,0] newFrame=[0,36][1200,1920] result=0x7 surface={Surface(name=null)/@0xafdd652 isValid=true -472643584}
01-24 08:56:56.444 28770 28770 D GeckoToolbar: onTabChanged: TITLE
01-24 08:56:56.445 28770 28770 D GeckoBrowserApp: BrowserApp.onTabChanged: 0: TITLE
01-24 08:56:56.445 28770 28770 D GeckoBrowserApp: onLocaleReady: de_DE
01-24 08:56:56.462 28770 28770 D GeckoNetworkManager: Incoming event receivedUpdate for state OnNoListeners -> OnNoListeners
01-24 08:56:56.463 28770 28770 D GeckoNetworkManager: New network state: UP, WIFI, WIFI
01-24 08:56:56.467 28770 28770 D GeckoToolbar: onTabChanged: MENU_UPDATED
01-24 08:56:56.467 28770 28770 D GeckoBrowserApp: BrowserApp.onTabChanged: 0: MENU_UPDATED
01-24 08:56:56.467 28770 28770 D GeckoToolbar: onTabChanged: MENU_UPDATED
01-24 08:56:56.467 28770 28770 D GeckoBrowserApp: BrowserApp.onTabChanged: 1: MENU_UPDATED
01-24 08:56:56.478 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: Relayout returned: oldFrame=[0,0][1200,1920] newFrame=[0,0][1200,1920] result=0x1 surface={isValid=true -472647680} surfaceGenerationChanged=false
01-24 08:56:56.515 28770 28780 W art     : Suspending all threads took: 24.987ms
01-24 08:56:56.523 28770 28780 I art     : Background sticky concurrent mark sweep GC freed 6558(425KB) AllocSpace objects, 4(120KB) LOS objects, 11% free, 4MB/4MB, paused 29.222ms total 48.731ms
01-24 08:56:56.532 28770 28770 D SurfaceView: Relayout returned: oldFrame=[0,36][1200,1920] newFrame=[0,36][1200,1920] result=0x1 surface={Surface(name=null)/@0xafdd652 isValid=true -472645632}
01-24 08:56:56.637 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: MSG_RESIZED_REPORT: frame=Rect(0, 0 - 1200, 1920) ci=Rect(0, 36 - 0, 0) vi=Rect(0, 36 - 0, 0) or=1
01-24 08:56:56.637 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: MSG_WINDOW_FOCUS_CHANGED 1
01-24 08:56:56.637 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: mHardwareRenderer.initializeIfNeeded()#2 mSurface={isValid=true -472647680}
01-24 08:56:56.643 28770 28770 I InputMethodManager: [IMM] startInputInner - mService.startInputOrWindowGainedFocus
01-24 08:56:56.651 28770 28783 D InputTransport: Input channel constructed: fd=70
01-24 08:56:56.653 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: Relayout returned: oldFrame=[0,0][1200,1920] newFrame=[0,0][1200,1920] result=0x1 surface={isValid=true -472647680} surfaceGenerationChanged=false
01-24 08:56:58.282 28770 28790 D SwitchBoard: Result: {"data":[{"schema":1512502894294,"buckets":{"max":"49","min":"0"},"match":{"appId":"^org.mozilla.firefox_beta$|^org.mozilla.firefox$","lang":"eng|zho|deu","regions":[]},"description":"Enable Leanplum SDK - Bug 1351571 \nExpand English Users to more region - Bug 1411066\nEnable  50% eng|zho|deu globally for Leanplum. see https://bugzilla.mozilla.org/show_bug.cgi?id=1411066#c8","id":"12f8f0dc-6401-402e-9e7d-3aec52576b87","name":"leanplum-start","last_modified":1512553130595},{"schema":1510207707840,"buckets":{"max":"100","min":"0"},"match":{"regions":[]},"description":"Allows apps to open tabs in a customized UI.","id":"5e23b482-8800-47be-b6dc-1a3bb6e455d4","name":"custom-tabs","last_modified":1510211043874},{"schema":1498764179980,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$","regions":[]},"description":"Enable Activity stream by default for users in the \"opt out\" group.","id":"7d504093-67c4-4afb-adf5-5ad23c7c1995","name":"activity-stream-opt-out","last_modified":1500969355986},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$"},"description":"Bug 1232439 - Show full-page edit bookmark dialog","id":"9ae1019b-9107-47c5-83f3-afa73360b020","name":"full-bookmark-management","last_modified":1498690258010},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{},"description":"Show addon menu item in top-level.","id":"46894232-177a-4cd1-b620-47c0b8e5e2aa","name":"top-addons-menu","last_modified":1498599522440},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"last_modified":1497643056372,"match":{"appId":"^org.mozilla.fennec|org.mozilla.firefox_beta"},"id":"5e4277e0-1029-ea14-1b74-5d25d301c5dc","name":"offline-cache"},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{},"description":"Gate flag for controlling if background telemetry processing (sync ping) is enabled or not.","id":"e6f9d217-3f43-478f-bff3-7829d7b9eeeb","name":"process-background-telemetry","last_modified":1496971360625},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$"},"description":"Show a setting in \"experimental features\" for enabling/disabling activity stream.","id":"7a022463-67fd-4ba3-8b06-a79d0c5e1fdc","name":"activity-stream-setting","last_modified":1496331790186},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$"},"description":"Enable/Disable Activity Stream","id":"d4fd9cfb-4c8b-4963-b21e-1c2f4bcd61d6","name":"activity-stream","last_modified":1496331773809},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"last_modified":1485853244635,"match":{"appId":""},"id":"4d2fa5c3-18b2-8734-49be-fe58993d2cf6","name":"download-content-catalog-sync"},{"schema":1480618438089,"buckets":{"max":"100","min":"50"},"last_modified":1482264639326,"match":{"appId":"^org.mozilla.fennec.*$|^org.mozilla.firefox_beta$"},"values":{"minimumTotalVisits":5,"lastVisitMaximumAgeMs":600000,"lastVisitMinimumAgeMs":30000},"id":"1d05fa3e-095f-b29a-d9b6-ab3a578efd0b","name":"promote-add-to-homescreen"},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"last_modified":1482262302021,"match":{"appId":"^org.mozilla.fennec.*$|^org.mozilla.firefox_beta$"},"id":"02d7caa1-cd9e-6949-084c-18bc9d468b6b","name":"triple-readerview-bookmark-prompt"},{"schema":1480618438089,"buckets":{"max":"50","min":"0"},"match":{},"description":"Arrange tabs in two columns in portrait mode (tabs tray)","id":"14fdc9f3-cf11-4bee-84f6-98495d08c61f","name":"compact-tabs","last_modified":1482242613284},{"schema":1467794476773,"buckets":{"max":"100","min":"0"},"last_modified":1477907551487,"match":{},"id":"d9f9f124-a4d6-47db-a9f4-cf0d00915088","name":"hls-video-playback"},{"buckets":{"max":"100","min":"0"},"id":"9a53ebfa-772d-d2d8-8307-f98943310360","match":{},"name":"bookmark-history-menu","last_modified":1467794477013},{"buckets":{"max":"0","min":"0"},"id":"3e4cef10-3a87-3cdd-4562-0062c2a9125b","match":{},"name":"content-notifications-12hrs
01-24 08:56:58.284 28770 28790 I System.out: (HTTPLog)-Static: isSBSettingEnabled false
01-24 08:56:58.284 28770 28790 I System.out: (HTTPLog)-Static: isSBSettingEnabled false
01-24 08:56:58.351 28770 28790 D SwitchBoard: Result: {"data":[{"schema":1512502894294,"buckets":{"max":"49","min":"0"},"match":{"appId":"^org.mozilla.firefox_beta$|^org.mozilla.firefox$","lang":"eng|zho|deu","regions":[]},"description":"Enable Leanplum SDK - Bug 1351571 \nExpand English Users to more region - Bug 1411066\nEnable  50% eng|zho|deu globally for Leanplum. see https://bugzilla.mozilla.org/show_bug.cgi?id=1411066#c8","id":"12f8f0dc-6401-402e-9e7d-3aec52576b87","name":"leanplum-start","last_modified":1512553130595},{"schema":1510207707840,"buckets":{"max":"100","min":"0"},"match":{"regions":[]},"description":"Allows apps to open tabs in a customized UI.","id":"5e23b482-8800-47be-b6dc-1a3bb6e455d4","name":"custom-tabs","last_modified":1510211043874},{"schema":1498764179980,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$","regions":[]},"description":"Enable Activity stream by default for users in the \"opt out\" group.","id":"7d504093-67c4-4afb-adf5-5ad23c7c1995","name":"activity-stream-opt-out","last_modified":1500969355986},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$"},"description":"Bug 1232439 - Show full-page edit bookmark dialog","id":"9ae1019b-9107-47c5-83f3-afa73360b020","name":"full-bookmark-management","last_modified":1498690258010},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{},"description":"Show addon menu item in top-level.","id":"46894232-177a-4cd1-b620-47c0b8e5e2aa","name":"top-addons-menu","last_modified":1498599522440},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"last_modified":1497643056372,"match":{"appId":"^org.mozilla.fennec|org.mozilla.firefox_beta"},"id":"5e4277e0-1029-ea14-1b74-5d25d301c5dc","name":"offline-cache"},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{},"description":"Gate flag for controlling if background telemetry processing (sync ping) is enabled or not.","id":"e6f9d217-3f43-478f-bff3-7829d7b9eeeb","name":"process-background-telemetry","last_modified":1496971360625},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$"},"description":"Show a setting in \"experimental features\" for enabling/disabling activity stream.","id":"7a022463-67fd-4ba3-8b06-a79d0c5e1fdc","name":"activity-stream-setting","last_modified":1496331790186},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"match":{"appId":"^org.mozilla.fennec.*$"},"description":"Enable/Disable Activity Stream","id":"d4fd9cfb-4c8b-4963-b21e-1c2f4bcd61d6","name":"activity-stream","last_modified":1496331773809},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"last_modified":1485853244635,"match":{"appId":""},"id":"4d2fa5c3-18b2-8734-49be-fe58993d2cf6","name":"download-content-catalog-sync"},{"schema":1480618438089,"buckets":{"max":"100","min":"50"},"last_modified":1482264639326,"match":{"appId":"^org.mozilla.fennec.*$|^org.mozilla.firefox_beta$"},"values":{"minimumTotalVisits":5,"lastVisitMaximumAgeMs":600000,"lastVisitMinimumAgeMs":30000},"id":"1d05fa3e-095f-b29a-d9b6-ab3a578efd0b","name":"promote-add-to-homescreen"},{"schema":1480618438089,"buckets":{"max":"100","min":"0"},"last_modified":1482262302021,"match":{"appId":"^org.mozilla.fennec.*$|^org.mozilla.firefox_beta$"},"id":"02d7caa1-cd9e-6949-084c-18bc9d468b6b","name":"triple-readerview-bookmark-prompt"},{"schema":1480618438089,"buckets":{"max":"50","min":"0"},"match":{},"description":"Arrange tabs in two columns in portrait mode (tabs tray)","id":"14fdc9f3-cf11-4bee-84f6-98495d08c61f","name":"compact-tabs","last_modified":1482242613284},{"schema":1467794476773,"buckets":{"max":"100","min":"0"},"last_modified":1477907551487,"match":{},"id":"d9f9f124-a4d6-47db-a9f4-cf0d00915088","name":"hls-video-playback"},{"buckets":{"max":"100","min":"0"},"id":"9a53ebfa-772d-d2d8-8307-f98943310360","match":{},"name":"bookmark-history-menu","last_modified":1467794477013},{"buckets":{"max":"0","min":"0"},"id":"3e4cef10-3a87-3cdd-4562-0062c2a9125b","match":{},"name":"content-notifications-12hrs
01-24 08:56:58.361 28770 28790 D SwitchBoard: leanplum-start = false
01-24 08:57:02.593 28770 28788 E GeckoLinker: XZ decoding: corrupt input stream
01-24 08:57:02.593 28770 28788 E GeckoLinker: Error decoding XZ file /data/user/0/org.mozilla.firefox/cache/libxul.so
01-24 08:57:02.908 28770 28788 E GeckoLibLoad: Couldn't get a handle to libxul!
01-24 08:57:02.908 28770 28788 E GeckoLibLoad: Throw
01-24 08:57:02.911 28770 28788 W GeckoThread: Clearing cache after load libs exception
01-24 08:57:02.911 28770 28788 W GeckoThread: java.lang.Exception: Error loading gecko libraries
01-24 08:57:02.911 28770 28788 W GeckoThread: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)
01-24 08:57:02.911 28770 28788 W GeckoThread: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibs(GeckoLoader.java:464)
01-24 08:57:02.911 28770 28788 W GeckoThread: 	at org.mozilla.gecko.GeckoThread.loadGeckoLibs(GeckoThread.java:247)
01-24 08:57:02.911 28770 28788 W GeckoThread: 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:269)
01-24 08:57:02.911 28770 28788 W GeckoThread: 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:365)
01-24 08:57:02.929 28770 28787 D GeckoJarReader: No Entry for chrome/de-DE/locale/de-DE/browser/searchplugins/list.json
01-24 08:57:02.961 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: ViewPostImeInputStage processKey 0
01-24 08:57:02.965 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: ViewPostImeInputStage processKey 1
01-24 08:57:02.984 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: MSG_WINDOW_FOCUS_CHANGED 0
01-24 08:57:03.229 28770 28770 D SurfaceView: Relayout returned: oldFrame=[0,36][1200,1920] newFrame=[0,36][1200,1920] result=0x5 surface={Surface(name=null)/@0xafdd652 isValid=false 0}
01-24 08:57:03.375 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: mHardwareRenderer.destroy()#1
01-24 08:57:03.386 28770 28770 D ViewRootImpl@c8e1548[BrowserApp]: Relayout returned: oldFrame=[0,0][1200,1920] newFrame=[0,0][1200,1920] result=0x5 surface={isValid=false 0} surfaceGenerationChanged=true
01-24 08:57:04.082 28770 28770 D InputTransport: Input channel destroyed: fd=70
01-24 08:57:04.160 28770 28770 D GeckoNetworkManager: Incoming event stop for state OnNoListeners -> OffNoListeners
01-24 08:57:04.182 28770 28770 D GeckoMemoryMonitor: onTrimMemory() notification received with level 20
01-24 08:57:04.182 28770 28770 D GeckoMemoryMonitor: increasing memory pressure to 1
01-24 08:57:08.708 28770 28788 E GeckoLinker: XZ decoding: corrupt input stream
01-24 08:57:08.708 28770 28788 E GeckoLinker: Error decoding XZ file /data/user/0/org.mozilla.firefox/cache/libxul.so
01-24 08:57:08.761 28770 28788 E GeckoLibLoad: Couldn't get a handle to libxul!
01-24 08:57:08.761 28770 28788 E GeckoLibLoad: Throw
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 84644 ("Gecko")
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: java.lang.Exception: Error loading gecko libraries
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibs(GeckoLoader.java:464)
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.loadGeckoLibs(GeckoThread.java:247)
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:279)
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:365)
01-24 08:57:08.761 28770 28788 E GeckoCrashHandler: Main thread (1) stack:
01-24 08:57:08.762 28770 28788 E GeckoCrashHandler:     android.os.MessageQueue.nativePollOnce(Native Method)
01-24 08:57:08.763 28770 28788 E GeckoCrashHandler:     android.os.MessageQueue.next(MessageQueue.java:323)
01-24 08:57:08.763 28770 28788 E GeckoCrashHandler:     android.os.Looper.loop(Looper.java:136)
01-24 08:57:08.763 28770 28788 E GeckoCrashHandler:     android.app.ActivityThread.main(ActivityThread.java:6776)
01-24 08:57:08.763 28770 28788 E GeckoCrashHandler:     java.lang.reflect.Method.invoke(Native Method)
01-24 08:57:08.763 28770 28788 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1496)
01-24 08:57:08.763 28770 28788 E GeckoCrashHandler:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1386)
01-24 08:57:08.763 28770 28788 E art     : No implementation found for void org.mozilla.gecko.GeckoAppShell.reportJavaCrash(java.lang.Throwable, java.lang.String) (tried Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash and Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash__Ljava_lang_Throwable_2Ljava_lang_String_2)

Comment 9

[James Willcox (:snorp) ([e-mail]) (he/him)]

Glandium, any ideas on this? See comments #7 and #8

Comment 10

[Mike Hommey [:glandium]]

Why are we trying to xz decode the cached libxul, which is presumably unxz'ed?

Comment 11

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Mike Hommey [:glandium] from comment #10)
> Why are we trying to xz decode the cached libxul, which is presumably
> unxz'ed?

It's not, I think that error message is just using the wrong file name.

Comment 12

[Mike Hommey [:glandium]]

Indeed, the code that displays that message is using the cache file path. It /could/ print something better, but it doesn't have access to the path of the file within the zip.

Anyways, this pretty much smells like corrupted APKs... presumably, android should barf about that(?), so are they corrupted at the source?

Comment 13

[Eugen Sawin [:esawin]]

This looks like APK corruption, the input stream is truncated or otherwise corrupt.
I think, we could also hit this error if we're low on space but the system does not return an error on ftruncate.

Looking at the crash stats, most affected phones are Samsung J7 and Blackberries (Passport, Z10), which is an usual distribution given the rarity of these devices. I assume this to be a device-specific issue.

Comment 14

[(inactive) Jim Chen [:jchen] [:darchons]]

I have patches to show a notification to reinstall the app if APK corruption is detected (based on CRC mismatch)

Comment 15

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Bug 1391268 - 1. Add call to verify CRC;

To reliably detect corrupt APK, this patch adds a MOZ_LINKER_VERIFY_CRC
env var to enable verification of CRC before decompressing XZ streams.

Review commit: https://reviewboard.mozilla.org/r/216272/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/216272/

Comment 16

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Bug 1391268 - 2. Expose mozglue linker error message to Java code;

This patch makes us save the last message from an ERROR() statement, and
use that message when throwing an exception to Java code. This does make
ERROR() not thread-safe, but I think that is okay since it's only used
by the single-threaded linker.

Review commit: https://reviewboard.mozilla.org/r/216274/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/216274/

Comment 17

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Bug 1391268 - 3. Detect corrupt APK when loading libs;

On the second pass after failing the first time to load libraries,
enable CRC verification and detect if the APK is corrupt. In that case,
instead of crashing, we send out an event to the application.

Review commit: https://reviewboard.mozilla.org/r/216276/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/216276/

Comment 18

[(inactive) Jim Chen [:jchen] [:darchons]]

Attachment: Bug 1391268 - 4. Display message when APK is corrupt;

Display a message on startup when the APK is corrupt. Right now the
"wrong SDK" message is displayed. I think we can potentially reuse that
message for Beta, but we need a different message for Nightly.

Review commit: https://reviewboard.mozilla.org/r/216278/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/216278/

Comment 19

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946305 [details]
Bug 1391268 - 4. Display message when APK is corrupt;

:antlam, does the corrupt_apk message sound okay? e.g. "This Firefox installation is corrupt. Please reinstall."

Comment 20

[Mike Hommey [:glandium]]

Comment on attachment 8946302 [details]
Bug 1391268 - 1. Add call to verify CRC;

https://reviewboard.mozilla.org/r/216272/#review222262

It seems to me you're going through hoops here, especially with the second and third patch, where you rely on checking whether some string appears in the last error the linker printed out. If the first time failed and we expect a crc mismatch, we're obviously not expecting library loading to work. Why not, instead of retrying through the linker, do a full scan of the apk in a jni function that iterates over the Zip file and does a crc check of all the files? Possibly, this could do other checks (like, in the case of XZ streams, check the streams)

Comment 21

[(inactive) Jim Chen [:jchen] [:darchons]]

During extraction seemed like a good place to check CRC, but fair enough.

I do still think it's a good idea to expose the error messages through the exception (i.e. part 2). "Error loading gecko libraries" is not a very good crash message.

Comment 22

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8946304 [details]
Bug 1391268 - 3. Detect corrupt APK when loading libs;

https://reviewboard.mozilla.org/r/216276/#review222516

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoThread.java:57
(Diff revision 1)
>          // After granting request to shutdown
>          @WrapForJNI EXITING(3),
>          // After granting request to restart
>          @WrapForJNI RESTARTING(3),
> +        // After failed lib extraction due to corrupted APK
> +                    CORRUPT_APK(2),

Fix indentation

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoThread.java:290
(Diff revision 1)
> -            FileUtils.delTree(GeckoLoader.getCacheDir(context),
> +        FileUtils.delTree(GeckoLoader.getCacheDir(context),
> -                              new FileUtils.FilenameRegexFilter(".*\\.so(?:\\.crc)?$"),
> +                          new FileUtils.FilenameRegexFilter(".*\\.so(?:\\.crc)?$"),
> -                              /* recurse */ true);
> +                          /* recurse */ true);
>  
> -            // Then try loading again. If this throws again, we actually crash.
> +        // Then try loading again. If this throws again, we either show
> +        // a corrupt apk message or actually crash.

I think I agree with Glandium here that it may be more useful to do some verification of the APK instead of just trying to load again.

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoThread.java:296
(Diff revision 1)
> +        try {
> +            GeckoLoader.setVerifyCRC(true);
>              loadGeckoLibs(context, resourcePath);
> +            return;
> +        } catch (final Exception e) {
> +            if (e.getMessage() != null && e.getMessage().contains("CRC mismatch")) {

Is there really any reason this could fail? Maybe if you have x86 APK installed on ARM? I think you could probably treat all errors here as "app install is hosed".

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoThread.java:307
(Diff revision 1)
> +            }
> +        }
> -        }
> +    }
>  
> -        setState(State.LIBS_READY);
> +    private static void notifyCorruptAPK() {
> +        EventDispatcher.getInstance().dispatch("Gecko:CorruptAPK", null);

This seems weird. I think we should just let the app install an exit listener for the GeckoThread, and if it exits with the CORRUPTED state it can show an error.

Comment 23

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8946305 [details]
Bug 1391268 - 4. Display message when APK is corrupt;

https://reviewboard.mozilla.org/r/216278/#review222530

::: mobile/android/base/java/org/mozilla/gecko/GeckoApp.java:660
(Diff revision 1)
>                  public void run() {
>                      getSharedPreferences().edit().putInt(PREFS_CRASHED_COUNT, 0).apply();
>                  }
>              }, STARTUP_PHASE_DURATION_MS);
>  
> +        } else if (event.equals("Gecko:CorruptAPK")) {

As I said in the other patch, I think it would be a lot better to just have some error handler directly on GeckoThread

Comment 24

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Jim Chen [:jchen] [:darchons] from comment #21)
> During extraction seemed like a good place to check CRC, but fair enough.

If we suspect the APK is corrupted we should check the zip checksums instead of the XZ ones, right?

Comment 25

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Comment on attachment 8946305 [details]
Bug 1391268 - 4. Display message when APK is corrupt;

Talked to jchen on Slack, we think the following message makes the most sense:

Unable to open Firefox. Please reinstall and try again.

Comment 26

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946302 [details]
Bug 1391268 - 1. Add call to verify CRC;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/216272/diff/1-2/

Comment 27

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946303 [details]
Bug 1391268 - 2. Expose mozglue linker error message to Java code;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/216274/diff/1-2/

Comment 28

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946304 [details]
Bug 1391268 - 3. Detect corrupt APK when loading libs;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/216276/diff/1-2/

Comment 29

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946305 [details]
Bug 1391268 - 4. Display message when APK is corrupt;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/216278/diff/1-2/

Comment 30

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946302 [details]
Bug 1391268 - 1. Add call to verify CRC;

https://reviewboard.mozilla.org/r/216272/#review222262

Changed part 1 to add a JNI method that verifies CRC checksums for a zip file. I kept part 2 because I still think it's valuable to have a more detailed crash message when we fail to load libs, instead of a generic "cannot load gecko libs" message.

Comment 31

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946304 [details]
Bug 1391268 - 3. Detect corrupt APK when loading libs;

https://reviewboard.mozilla.org/r/216276/#review222516

> Is there really any reason this could fail? Maybe if you have x86 APK installed on ARM? I think you could probably treat all errors here as "app install is hosed".

I think we should still throw on anything that's not corrupted APK, so we don't hide linker bugs, for example

> This seems weird. I think we should just let the app install an exit listener for the GeckoThread, and if it exits with the CORRUPTED state it can show an error.

So we can't actually exit GeckoThread because GeckoApplication kills the app if we exit GeckoThread. So we hold GeckoThread in a suspended state and send out an event.

Comment 32

[Mike Hommey [:glandium]]

Comment on attachment 8946302 [details]
Bug 1391268 - 1. Add call to verify CRC;

https://reviewboard.mozilla.org/r/216272/#review222864

::: mozglue/linker/Zip.h:269
(Diff revision 2)
>    }
>  
> +  /**
> +   * Returns whether all files have correct CRC checksum.
> +   */
> +  bool VerifyCRC() const;

CRCs (plural), maybe?

::: mozglue/linker/Zip.cpp:208
(Diff revision 2)
> +      crc = crc32(crc, static_cast<const uint8_t*>(file->GetData()),
> +                  entry->compressedSize);
> +      DEBUG_LOG(" STORE size=%d crc=%08x", int(entry->compressedSize), crc);
> +
> +    } else if (entry->compression == Stream::Type::DEFLATE) {
> +      zxx_stream zstream;

It would be easier, albeit less cpu cache efficient, and more memory-hungre, to use MappableDeflate::mmap.

::: mozglue/linker/Zip.cpp:236
(Diff revision 2)
> +
> +      inflateEnd(&zstream);
> +      DEBUG_LOG(" DEFLATE size=%d crc=%08x", int(zstream.total_out), crc);
> +
> +    } else {
> +      continue;

maybe MOZ_ASSERT_UNREACHABLE() here?

Comment 33

[Mike Hommey [:glandium]]

Comment on attachment 8946303 [details]
Bug 1391268 - 2. Expose mozglue linker error message to Java code;

https://reviewboard.mozilla.org/r/216274/#review222866

I'm still not convinced here. Do you want to expose this to users? Is that giving them any usefulness? I can understand it would give /us/ some more information to go by, but then, we can just look at logcat. Maybe we need an easier way to get logcat information from users, but that would be a more general problem, and even for the linker, the last error may well be not enough ; more context would be more useful in many cases.

Comment 34

[(inactive) Jim Chen [:jchen] [:darchons]]

Comment on attachment 8946303 [details]
Bug 1391268 - 2. Expose mozglue linker error message to Java code;

https://reviewboard.mozilla.org/r/216274/#review222866

Yeah it's to give us more information on crash-stats, considering "load gecko libs" involves loading multiple libraries each with many steps. These crashes all have the same Java stack, so the stack is pretty useless.

Crash reports with logcat are few and far between, and that's by-design because the user has to opt-in to include logcat. Besides, not everyone has access to logcat from crash-stats. IMO if logcat is the only way for us to determine where an error occurred, we need to do something better.

Comment 35

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8946304 [details]
Bug 1391268 - 3. Detect corrupt APK when loading libs;

https://reviewboard.mozilla.org/r/216276/#review222978

Comment 36

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8946305 [details]
Bug 1391268 - 4. Display message when APK is corrupt;

https://reviewboard.mozilla.org/r/216278/#review223002

Comment 37

[(inactive) Jim Chen [:jchen] [:darchons]]

Does comment 34 sound reasonable? I think it makes a lot of sense to have more detailed crash message on crash-stats for these lib-loading crashes. If there are ways to improve this patch, let me know.

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/08726d7c7b2c
1. Add call to verify CRC; r=glandium
https://hg.mozilla.org/integration/mozilla-inbound/rev/564e277ecce1
3. Detect corrupt APK when loading libs; r=snorp
https://hg.mozilla.org/integration/mozilla-inbound/rev/5adf396b4503
4. Display message when APK is corrupt; r=snorp

Comment 39

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/08726d7c7b2c
https://hg.mozilla.org/mozilla-central/rev/564e277ecce1
https://hg.mozilla.org/mozilla-central/rev/5adf396b4503

Comment 40

[Nick Alexander :nalexander [he/him]]

I notice looking at the matching crash reports that in 59.0b*, the most represented devices (Z10 and Passport) appear to say "nothumb" in their device descriptions.  Is it possible that we're simply trying to load libraries that use an invalid instruction set on some devices?

Comment 41

[Eugen Sawin [:esawin]]

Broken Thumb support could explain the corruption during the libxul extraction.
This should be easy to test by disabling the BCJ filter in https://searchfox.org/mozilla-central/source/modules/xz-embedded/moz.build#27.

We would need a reproducing device for that (e.g., Blackberry Passport or Z10).

Comment 42

[Eugen Sawin [:esawin]]

We need to disable the filter during compression, too, in https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/action/package_fennec_apk.py#106 .
Or just disable Thumb support overall in https://searchfox.org/mozilla-central/source/build/autoconf/arch.m4#32 .

Comment 43

[(inactive) Jim Chen [:jchen] [:darchons]]

(In reply to Nick Alexander :nalexander from comment #40)
> I notice looking at the matching crash reports that in 59.0b*, the most
> represented devices (Z10 and Passport) appear to say "nothumb" in their
> device descriptions.  Is it possible that we're simply trying to load
> libraries that use an invalid instruction set on some devices?

Are you referring to the "App Notes" in crash reports? AFAICT all crash reports say "nothumb Build", which comes from [1], because of the faulty logic at [2] (the actual switch is "-march=armv7-a"). It's very misleading. We should probably remove that altogether.

[1] https://searchfox.org/mozilla-central/rev/2ce99e8054b0ff6ed1adf484aeaacacf2fea084c/mobile/android/base/java/org/mozilla/gecko/CrashReporter.java#512
[2] https://searchfox.org/mozilla-central/rev/2ce99e8054b0ff6ed1adf484aeaacacf2fea084c/mobile/android/base/generate_build_config.py#101

(In reply to Eugen Sawin [:esawin] from comment #41)
> This should be easy to test by disabling the BCJ filter in
> https://searchfox.org/mozilla-central/source/modules/xz-embedded/moz.
> build#27.

The BCJ filter shouldn't cause corruption by itself. You can use the x86 filter on thumb code and it would still work, just not compress as well. But it would be a problem if we're doing something like filtering with thumb and then unfiltering with x86.

Comment 44

[Nick Alexander :nalexander [he/him]]

(In reply to Jim Chen [:jchen] [:darchons] from comment #43)
> (In reply to Nick Alexander :nalexander from comment #40)
> > I notice looking at the matching crash reports that in 59.0b*, the most
> > represented devices (Z10 and Passport) appear to say "nothumb" in their
> > device descriptions.  Is it possible that we're simply trying to load
> > libraries that use an invalid instruction set on some devices?
> 
> Are you referring to the "App Notes" in crash reports? AFAICT all crash
> reports say "nothumb Build", which comes from [1], because of the faulty
> logic at [2] (the actual switch is "-march=armv7-a"). It's very misleading.
> We should probably remove that altogether.

I was, and the current situation is terrible :/

> [1]
> https://searchfox.org/mozilla-central/rev/
> 2ce99e8054b0ff6ed1adf484aeaacacf2fea084c/mobile/android/base/java/org/
> mozilla/gecko/CrashReporter.java#512
> [2]
> https://searchfox.org/mozilla-central/rev/
> 2ce99e8054b0ff6ed1adf484aeaacacf2fea084c/mobile/android/base/
> generate_build_config.py#101
> 
> (In reply to Eugen Sawin [:esawin] from comment #41)
> > This should be easy to test by disabling the BCJ filter in
> > https://searchfox.org/mozilla-central/source/modules/xz-embedded/moz.
> > build#27.
> 
> The BCJ filter shouldn't cause corruption by itself. You can use the x86
> filter on thumb code and it would still work, just not compress as well. But
> it would be a problem if we're doing something like filtering with thumb and
> then unfiltering with x86.

It sounds like the team is well aware of any issues that might arise with thumb builds, so there's nothing actionable here.  Thanks esawin, thanks jchen.

Comment 45

[James Willcox (:snorp) ([e-mail]) (he/him)]

I was looking through some of the crash reports with logcats today and found this:

04-09 19:28:43.651 17555700    13 E GeckoLinker: /data/app/org.mozilla.firefox-1.apk!/assets/armeabi-v7a/libxul.so: Relocation to NULL @0x03219f90 for symbol "isnanf"
04-09 19:28:43.662 17555700    13 E GeckoLibLoad: Couldn't get a handle to libxul!
04-09 19:28:43.662 17555700    13 E GeckoLibLoad: Throw
04-09 19:28:43.663 17555700    13 W GeckoThread: Clearing cache after load libs exception
04-09 19:28:43.663 17555700    13 W GeckoThread: java.lang.Exception: Error loading gecko libraries
04-09 19:28:43.663 17555700    13 W GeckoThread: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)
04-09 19:28:43.663 17555700    13 W GeckoThread: 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibs(GeckoLoader.java:468)
04-09 19:28:43.663 17555700    13 W GeckoThread: 	at org.mozilla.gecko.GeckoThread.loadGeckoLibs(GeckoThread.java:251)
04-09 19:28:43.663 17555700    13 W GeckoThread: 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:273)
04-09 19:28:43.663 17555700    13 W GeckoThread: 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:369)

In this case it looks like loading libxul actually succeeded, because CustomElf appears to be printing errors for all failure cases, but the dlopen() failed. I think this particular case may be a dup of bug 1447607. I'll request uplift for that and see if it helps at all.

Comment 46

[Mike Hommey [:glandium]]

The first line you quote actually says exactly what the problem is, and it's not related to bug 1447607: libxul.so needs the isnanf symbol, and the linker can't find it.

Comment 47

[(inactive) Jim Chen [:jchen] [:darchons]]

The NI was about something else (comment 37), but this is a good example of what attachment 8946303 [details] is trying to do. With that, this missing symbol message would have been very apparent in the crash report, rather than requiring digging through logcat messages manually.

Comment 48

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Mike Hommey [:glandium] from comment #46)
> The first line you quote actually says exactly what the problem is, and it's
> not related to bug 1447607: libxul.so needs the isnanf symbol, and the
> linker can't find it.

Ah, I thought that was just a warning, but I see the same message is in fact an error as well. That's a bit confusing. It seems like I see relocation problems all the time that are not fatal.

crazy_linker has this problem as well and apparently the fix[0] is to just redirect to __isnanf instead.

[0] https://android.googlesource.com/platform/ndk/+/3ad0c56%5E!/

Comment 49

[James Willcox (:snorp) ([e-mail]) (he/him)]

Mike it looks like we already had a couple different fixes here, the most recent being https://hg.mozilla.org/mozilla-central/rev/339e44e41f22

Do you have an idea why that wouldn't be working here?

Comment 50

[Mike Hommey [:glandium]]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #48)
> (In reply to Mike Hommey [:glandium] from comment #46)
> > The first line you quote actually says exactly what the problem is, and it's
> > not related to bug 1447607: libxul.so needs the isnanf symbol, and the
> > linker can't find it.
> 
> Ah, I thought that was just a warning, but I see the same message is in fact
> an error as well. That's a bit confusing. It seems like I see relocation
> problems all the time that are not fatal.

Because there are relocation problems that are not fatal. Note the E before GeckoLinker instead of W, in the log.

(In reply to James Willcox (:snorp) ([e-mail]) from comment #49)
> Mike it looks like we already had a couple different fixes here, the most
> recent being https://hg.mozilla.org/mozilla-central/rev/339e44e41f22
> 
> Do you have an idea why that wouldn't be working here?

Not without a full MOZ_DEBUG_LINKER log unfortunately. And/or a copy of the corresponding libm.so. Has anyone from the team been able to reproduce yet?

Comment 51

[Mike Hommey [:glandium]]

(In reply to Jim Chen [:jchen] [:darchons] from comment #47)
> The NI was about something else (comment 37), but this is a good example of
> what attachment 8946303 [details] is trying to do. With that, this missing
> symbol message would have been very apparent in the crash report, rather
> than requiring digging through logcat messages manually.

Comment 33 still applies. I seems to me we should more generally wrap __android_log_print, and keep ANDROID_LOG_ERROR messages for some values of the second argument to that function, GeckoLinker being one of them.

Comment 52

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 59 and 60. I don't see any crashes for 60 in the last month. Was it really affected?

Comment 53

[Liz Henry (:lizzard) (relman/hg->git project)]

glandium, darchons, is this still a review in progress?  This crash is a very high volume on release but I can't tell if 60 and 61 are affected.

Comment 54

[(inactive) Jim Chen [:jchen] [:darchons]]

That's only a diagnostic patch. We don't have a fix for the remaining crashes yet unfortunately.

Comment 55

[dapgo]

error log from Firefox 60.0.1 in Android runtime 4.3 (BB10):

W/GeckoLinker(36036872): /data/app/org.mozilla.firefox-1.apk!/assets/armeabi-v7a/libxul.so: Relocation to NULL @0x0338c344
W/GeckoLinker(36036872): /data/app/org.mozilla.firefox-1.apk!/assets/armeabi-v7a/libxul.so: Relocation to NULL @0x0338c3c8
W/GeckoLinker(36036872): /data/app/org.mozilla.firefox-1.apk!/assets/armeabi-v7a/libxul.so: Relocation to NULL @0x0338c4cc
E/GeckoLinker(36036872): /data/app/org.mozilla.firefox-1.apk!/assets/armeabi-v7a/libxul.so: Relocation to NULL @0x0338f290 for symbol "isnanf"
E/GeckoLibLoad(36036872): Couldn't get a handle to libxul!
E/GeckoLibLoad(36036872): Throw
W/dalvikvm(36036872): threadid=11: thread exiting with uncaught exception (group=11abb970)
E/GeckoCrashHandler(36036872): >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 14 ("Gecko")
E/GeckoCrashHandler(36036872): java.lang.Exception: Error loading gecko libraries
E/GeckoCrashHandler(36036872): 	at org.mozilla.gecko.mozglue.GeckoLoader.loadGeckoLibsNative(Native Method)

Comment 56

[dapgo]

Just for your information, the FF fork Waterfox55 produces a warning in same lib but it doesn't crash:

W/GeckoLinker(65683721): /data/app/org.waterfoxproject.waterfox-1.apk!/assets/armeabi-v7a/libnss3.so: unhandled flags #8 not handled
W/GeckoLinker(65683721): /data/app/org.waterfoxproject.waterfox-1.apk!/assets/armeabi-v7a/libxul.so: unhandled flags #8 not handled
W/GeckoLinker(65683721): /data/app/org.waterfoxproject.waterfox-1.apk!/assets/armeabi-v7a/liblgpllibs.so: unhandled flags #8 not handled
W/ActivityThread(65683721): ClassLoader.loadClass: The class loader returned by Thread.getContextClassLoader() may fail for processes that host multiple applications. You should explicitly specify a context class loader. For example: Thread.setContextClassLoader(getClass().getClassLoader());

(In reply to dapgo from comment #55)

> error log from Firefox 60.0.1 in Android runtime 4.3 (BB10):
> .....
> E/GeckoLinker(36036872):
> /data/app/org.mozilla.firefox-1.apk!/assets/armeabi-v7a/libxul.so:
> Relocation to NULL @0x0338f290 for symbol "isnanf"
> E/GeckoLibLoad(36036872): Couldn't get a handle to libxul!
> E/GeckoLibLoad(36036872): Throw
> .....

Comment 57

[Firefox Bug Husbandry Bot]

Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.

Comment 58

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.