1314546 - Can't load native libraries in artifact builds ("Not a seekable zstream" / "Couldn't get a handle to libnss3!")

Assignee

Description

•

8 years ago

This only affects artifact builds, but not normal builds. Startup crash, stack trace is below, but the actual failing line seems to be:

mozglue/android/APKOpen.cpp - line 285:
  nss_handle = dlopenAPKLibrary(apkName, "libnss3.so");


Startup trace:

11-02 01:42:23.112 21315-21335/? D/GeckoLoader: Gecko environment env0: null
11-02 01:42:23.127 21315-21335/? E/GeckoLibLoad: Load sqlite start
11-02 01:42:23.128 21315-21335/? E/GeckoLinker: Not a seekable zstream
11-02 01:42:23.128 21315-21335/? E/GeckoLibLoad: Couldn't get a handle to libnss3!
11-02 01:42:23.128 21315-21335/? E/GeckoLibLoad: Throw
11-02 01:42:23.128 21315-21335/? E/GeckoLibLoad: Load sqlite done
11-02 01:42:23.129 21315-21335/? E/GeckoCrashHandler: >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 282 ("Gecko")
                                                      java.lang.Exception: Error loading sqlite libraries
                                                          at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibsNative(Native Method)
                                                          at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibs(GeckoLoader.java:222)
                                                          at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:398)
                                                          at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:496)
11-02 01:42:23.129 21315-21335/? E/GeckoCrashHandler: Main thread (1) stack:
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.Fragment.isSupportFragmentClass(Fragment.java:451)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentManagerImpl.onCreateView(FragmentManager.java:2252)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentController.onCreateView(FragmentController.java:120)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentActivity.dispatchFragmentsOnCreateView(FragmentActivity.java:356)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.BaseFragmentActivityHoneycomb.onCreateView(BaseFragmentActivityHoneycomb.java:31)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentActivity.onCreateView(FragmentActivity.java:79)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:777)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:727)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:858)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:821)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:861)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:821)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:861)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:821)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:518)
11-02 01:42:23.147 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:426)
11-02 01:42:23.147 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:377)
11-02 01:42:23.148 21315-21335/? E/GeckoCrashHandler:     android.support.v7.app.AppCompatDelegateImplV7.setContentView(AppCompatDelegateImplV7.java:280)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.support.v7.app.AppCompatActivity.setContentView(AppCompatActivity.java:140)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     org.mozilla.gecko.GeckoApp.onCreate(GeckoApp.java:1237)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     org.mozilla.gecko.BrowserApp.onCreate(BrowserApp.java:612)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.Activity.performCreate(Activity.java:6664)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1118)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2599)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2707)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.-wrap12(ActivityThread.java)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread$H.handleMessage(ActivityThread.java:1460)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.os.Handler.dispatchMessage(Handler.java:102)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.os.Looper.loop(Looper.java:154)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.main(ActivityThread.java:6077)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     java.lang.reflect.Method.invoke(Native Method)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:865)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:755)
11-02 01:42:23.152 21315-21335/? E/art: No implementation found for void org.mozilla.gecko.GeckoAppShell.reportJavaCrash(java.lang.Throwable, java.lang.String) (tried Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash and Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash__Ljava_lang_Throwable_2Ljava_lang_String_2)
11-02 01:42:23.152 21315-21335/? E/AndroidRuntime: FATAL EXCEPTION: Gecko
                                                   Process: org.mozilla.fennec_andrzejhunt, PID: 21315
                                                   java.lang.Exception: Error loading sqlite libraries
                                                       at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibsNative(Native Method)
                                                       at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibs(GeckoLoader.java:222)
                                                       at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:398)
                                                       at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:496)
11-02 01:42:23.174 586-4215/? W/ActivityManager:   Force finishing activity org.mozilla.fennec_andrzejhunt/org.mozilla.gecko.BrowserApp

Comment hidden (mozreview-request)

This appears to convince ElfLoader to load our (compressed?) so's the
right way for artifact builds too. I am completely unfamiliar in this area,
and this doesn't seem like a landable patch - but it's good enough for my
local development for now.

Review commit: https://reviewboard.mozilla.org/r/90028/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90028/

Sebastian Kaspari (:sebastian; :pocmo)

Comment 2

•

8 years ago

I see the same error. Looking at |mach artifact install| it seems like there's no matching binary available:
 
> 0:01.88 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on mozilla-central.
> 0:01.88 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on integration/fx-team.
> 0:02.92 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on integration/mozilla-inbound.
> 0:03.78 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on releases/mozilla-aurora.
> 0:04.70 Retrieving the last 50 pushheads starting with id 30899 on mozilla-central
> 0:08.21 Installing from remote pushhead 21153294d3a0bf7ce8e5e6bb1a0e684745b88fda on mozilla-central
> 0:08.21 Installing from https://queue.taskcluster.net/v1/task/WEw3--SjTYy0dOnJPHt3xQ/artifacts/public%2Fbuild%2Ftarget.apk
> 0:08.22 Installing from /Users/sebastian/.mozbuild/package-frontend/ebf15fca6ac3a7a0-public%2Fbuild%2Ftarget.apk
> 0:08.22 Installing from processed /Users/sebastian/.mozbuild/package-frontend/ebf15fca6ac3a7a0-public%2Fbuild%2Ftarget.apk.processed.jar

3e73fd638e687a4d7f46613586e5156b8e2af846 vs. 21153294d3a0bf7ce8e5e6bb1a0e684745b88fda

Eugen Sawin [:esawin]

Comment 3

•

8 years ago

Bug 1291424 disabled the on-demand decompression linker and bug 1307886 enabled XZ-compression of the shared libraries (see linked bugs for all related linker changes).

This is a somewhat expected fallout from the compression scheme changes (https://bugzilla.mozilla.org/show_bug.cgi?id=1307886#c12).
We definitely don't want to revert any of the changes since they provide great benefits, so let's fix artifact builds instead.

Andrzej Hunt :ahunt

Assignee

Comment 4

•

8 years ago

(In reply to Eugen Sawin [:esawin] from comment #3)
> Bug 1291424 disabled the on-demand decompression linker and bug 1307886
> enabled XZ-compression of the shared libraries (see linked bugs for all
> related linker changes).
> 
> This is a somewhat expected fallout from the compression scheme changes
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1307886#c12).
> We definitely don't want to revert any of the changes since they provide
> great benefits, so let's fix artifact builds instead.

As far as I can tell, the issue is that:
- We depend on Java frontend code setting an env var (MOZ_LINKER_EXTRACT=1), based on a build flag (see GeckoLoader.java)
- which causes our c++ code to follow the wrong path when loading shared libs, since the env var isn't set (see ElfLoader.cpp)
- Which is indirectly caused by different build variables being set while building artifacts, and building locally (MOZ_LINKER_EXTRACT is only set for $COMPILE_ENVIRONMENT builds, which isn't the case for artifact builds, even though the artifacts that are being used might have been built that way.

Would it be better to instead move the build-flag dependent check directly into our C++ code, so that artifacts consistently load the correct library type based on whatever build config was used, without the frontend build having to care about how exactly libraries are loaded?

I.e. in ElfLoader.cpp, could we "#ifdef MOZ_LINKER_EXTRACT" instead of having a conditional on "getenv("MOZ_LINKER_EXTRACT");"?

Eugen Sawin [:esawin]

Comment 5

•

8 years ago

See above comment.

Flags: needinfo?(mh+mozilla)

Mike Hommey [:glandium]

Comment 6

•

8 years ago

I'd rather not change the linker code for that. Just unconditionally putenv on the java end.

Flags: needinfo?(mh+mozilla)

Comment hidden (mozreview-request)

Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/90028/diff/1-2/

Sebastian Kaspari (:sebastian; :pocmo)

Updated

•

8 years ago

Assignee: nobody → ahunt

Status: NEW → ASSIGNED

Iteration: --- → 1.8

Priority: -- → P1

Whiteboard: [MobileAS]

Eugen Sawin [:esawin]

Comment 8

•

8 years ago

mozreview-review

Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

https://reviewboard.mozilla.org/r/90028/#review90388

With this the MOZ_LINKER_EXTRACT build flag becomes redundant, I think it would be better to make MOZ_LINKER_EXTRACT=1 the default for Android in old-configure.in instead.

Eugen Sawin [:esawin]

Updated

•

8 years ago

Attachment #8806650 - Flags: review?(mh+mozilla)

Mike Hommey [:glandium]

Comment 9

•

8 years ago

mozreview-review

Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

https://reviewboard.mozilla.org/r/90028/#review91166

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/mozglue/GeckoLoader.java:194
(Diff revision 2)
>          String forced_ondemand = System.getenv("MOZ_LINKER_ONDEMAND");
>          if (forced_ondemand == null) {
>              if ("HTC".equals(android.os.Build.MANUFACTURER) &&
>                  "HTC Vision".equals(android.os.Build.MODEL)) {
>                  putenv("MOZ_LINKER_ONDEMAND=0");
>              }
>          }

BTW, this can go away.

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/mozglue/GeckoLoader.java:202
(Diff revision 2)
> -        if (AppConstants.MOZ_LINKER_EXTRACT) {
> +        // Artifact builds might not have MOZ_LINKER_EXTRACT set due to their separate build config,
> +        // however the artifacts that they use will have been built using MOZ_LINKER_EXTRACT,
> +        // hence we need to pass MOZ_LINKER_EXTRACT to the linker regardless of actual build config.

I don't think this comment needs to mention artifact builds. The main point is that what Firefox for Android wants now is that the linker always extracts files, and that's how this is made to happen because, as of now, it's not what the linker does by default.

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/mozglue/GeckoLoader.java:202
(Diff revision 2)
>                  "HTC Vision".equals(android.os.Build.MODEL)) {
>                  putenv("MOZ_LINKER_ONDEMAND=0");
>              }
>          }
>  
> -        if (AppConstants.MOZ_LINKER_EXTRACT) {
> +        // Artifact builds might not have MOZ_LINKER_EXTRACT set due to their separate build config,

You can also remove the constant.

Attachment #8806650 - Flags: review?(mh+mozilla) → review+

Eugen Sawin [:esawin]

Comment 10

•

8 years ago

mozreview-review

Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

https://reviewboard.mozilla.org/r/90028/#review91280

Attachment #8806650 - Flags: review?(esawin) → review+

Comment hidden (mozreview-request)

Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/90028/diff/2-3/

Pulsebot

Comment 12

•

8 years ago

Pushed by ahunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/720cb3b359c9
Always putenv(MOZ_LINKER_EXTRACT) for Android r=esawin,glandium

Carsten Book [:Tomcat]

Comment 13

•

8 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/720cb3b359c9

Status: ASSIGNED → RESOLVED

Closed: 8 years ago

status-firefox52: affected → fixed

Resolution: --- → FIXED

Target Milestone: --- → Firefox 52

BMO Automation

Updated

•

5 years ago

Product: Firefox for Android → Firefox Build System

Target Milestone: Firefox 52 → mozilla52

Bugzilla

Quick Search

Can't load native libraries in artifact builds ("Not a seekable zstream" / "Couldn't get a handle to libnss3!")

Categories

(Firefox Build System :: Android Studio and Gradle Integration, defect, P1)

Tracking

(firefox52 fixed)

People

(Reporter: ahunt, Assigned: ahunt)

References

Details

(Whiteboard: [MobileAS])

Crash Data

Security

(public)

User Story

Attachments

(1 file)

Description

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Updated

Comment 8

Updated

Comment 9

Comment 10

Comment 11

Comment 12

Comment 13

Updated

Attachment

General

Description

File Name

Content Type