Can't load native libraries in artifact builds ("Not a seekable zstream" / "Couldn't get a handle to libnss3!")

RESOLVED FIXED in Firefox 52

Status

()

defect
P1
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: ahunt, Assigned: ahunt)

Tracking

Trunk
Firefox 52
Points:
---

Firefox Tracking Flags

(firefox52 fixed)

Details

(Whiteboard: [MobileAS])

Attachments

(1 attachment)

Assignee

Description

3 years ago
This only affects artifact builds, but not normal builds. Startup crash, stack trace is below, but the actual failing line seems to be:

mozglue/android/APKOpen.cpp - line 285:
  nss_handle = dlopenAPKLibrary(apkName, "libnss3.so");


Startup trace:

11-02 01:42:23.112 21315-21335/? D/GeckoLoader: Gecko environment env0: null
11-02 01:42:23.127 21315-21335/? E/GeckoLibLoad: Load sqlite start
11-02 01:42:23.128 21315-21335/? E/GeckoLinker: Not a seekable zstream
11-02 01:42:23.128 21315-21335/? E/GeckoLibLoad: Couldn't get a handle to libnss3!
11-02 01:42:23.128 21315-21335/? E/GeckoLibLoad: Throw
11-02 01:42:23.128 21315-21335/? E/GeckoLibLoad: Load sqlite done
11-02 01:42:23.129 21315-21335/? E/GeckoCrashHandler: >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 282 ("Gecko")
                                                      java.lang.Exception: Error loading sqlite libraries
                                                          at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibsNative(Native Method)
                                                          at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibs(GeckoLoader.java:222)
                                                          at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:398)
                                                          at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:496)
11-02 01:42:23.129 21315-21335/? E/GeckoCrashHandler: Main thread (1) stack:
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.Fragment.isSupportFragmentClass(Fragment.java:451)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentManagerImpl.onCreateView(FragmentManager.java:2252)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentController.onCreateView(FragmentController.java:120)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentActivity.dispatchFragmentsOnCreateView(FragmentActivity.java:356)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.BaseFragmentActivityHoneycomb.onCreateView(BaseFragmentActivityHoneycomb.java:31)
11-02 01:42:23.136 21315-21335/? E/GeckoCrashHandler:     android.support.v4.app.FragmentActivity.onCreateView(FragmentActivity.java:79)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:777)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.createViewFromTag(LayoutInflater.java:727)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:858)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:821)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:861)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:821)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflate(LayoutInflater.java:861)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.rInflateChildren(LayoutInflater.java:821)
11-02 01:42:23.137 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:518)
11-02 01:42:23.147 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:426)
11-02 01:42:23.147 21315-21335/? E/GeckoCrashHandler:     android.view.LayoutInflater.inflate(LayoutInflater.java:377)
11-02 01:42:23.148 21315-21335/? E/GeckoCrashHandler:     android.support.v7.app.AppCompatDelegateImplV7.setContentView(AppCompatDelegateImplV7.java:280)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.support.v7.app.AppCompatActivity.setContentView(AppCompatActivity.java:140)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     org.mozilla.gecko.GeckoApp.onCreate(GeckoApp.java:1237)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     org.mozilla.gecko.BrowserApp.onCreate(BrowserApp.java:612)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.Activity.performCreate(Activity.java:6664)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1118)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2599)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2707)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.-wrap12(ActivityThread.java)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread$H.handleMessage(ActivityThread.java:1460)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.os.Handler.dispatchMessage(Handler.java:102)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.os.Looper.loop(Looper.java:154)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     android.app.ActivityThread.main(ActivityThread.java:6077)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     java.lang.reflect.Method.invoke(Native Method)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:865)
11-02 01:42:23.151 21315-21335/? E/GeckoCrashHandler:     com.android.internal.os.ZygoteInit.main(ZygoteInit.java:755)
11-02 01:42:23.152 21315-21335/? E/art: No implementation found for void org.mozilla.gecko.GeckoAppShell.reportJavaCrash(java.lang.Throwable, java.lang.String) (tried Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash and Java_org_mozilla_gecko_GeckoAppShell_reportJavaCrash__Ljava_lang_Throwable_2Ljava_lang_String_2)
11-02 01:42:23.152 21315-21335/? E/AndroidRuntime: FATAL EXCEPTION: Gecko
                                                   Process: org.mozilla.fennec_andrzejhunt, PID: 21315
                                                   java.lang.Exception: Error loading sqlite libraries
                                                       at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibsNative(Native Method)
                                                       at org.mozilla.gecko.mozglue.GeckoLoader.loadSQLiteLibs(GeckoLoader.java:222)
                                                       at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:398)
                                                       at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:496)
11-02 01:42:23.174 586-4215/? W/ActivityManager:   Force finishing activity org.mozilla.fennec_andrzejhunt/org.mozilla.gecko.BrowserApp
Comment hidden (mozreview-request)
I see the same error. Looking at |mach artifact install| it seems like there's no matching binary available:
 
> 0:01.88 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on mozilla-central.
> 0:01.88 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on integration/fx-team.
> 0:02.92 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on integration/mozilla-inbound.
> 0:03.78 Attempting to find a pushhead containing 3e73fd638e687a4d7f46613586e5156b8e2af846 on releases/mozilla-aurora.
> 0:04.70 Retrieving the last 50 pushheads starting with id 30899 on mozilla-central
> 0:08.21 Installing from remote pushhead 21153294d3a0bf7ce8e5e6bb1a0e684745b88fda on mozilla-central
> 0:08.21 Installing from https://queue.taskcluster.net/v1/task/WEw3--SjTYy0dOnJPHt3xQ/artifacts/public%2Fbuild%2Ftarget.apk
> 0:08.22 Installing from /Users/sebastian/.mozbuild/package-frontend/ebf15fca6ac3a7a0-public%2Fbuild%2Ftarget.apk
> 0:08.22 Installing from processed /Users/sebastian/.mozbuild/package-frontend/ebf15fca6ac3a7a0-public%2Fbuild%2Ftarget.apk.processed.jar

3e73fd638e687a4d7f46613586e5156b8e2af846 vs. 21153294d3a0bf7ce8e5e6bb1a0e684745b88fda
Bug 1291424 disabled the on-demand decompression linker and bug 1307886 enabled XZ-compression of the shared libraries (see linked bugs for all related linker changes).

This is a somewhat expected fallout from the compression scheme changes (https://bugzilla.mozilla.org/show_bug.cgi?id=1307886#c12).
We definitely don't want to revert any of the changes since they provide great benefits, so let's fix artifact builds instead.
Assignee

Comment 4

3 years ago
(In reply to Eugen Sawin [:esawin] from comment #3)
> Bug 1291424 disabled the on-demand decompression linker and bug 1307886
> enabled XZ-compression of the shared libraries (see linked bugs for all
> related linker changes).
> 
> This is a somewhat expected fallout from the compression scheme changes
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1307886#c12).
> We definitely don't want to revert any of the changes since they provide
> great benefits, so let's fix artifact builds instead.

As far as I can tell, the issue is that:
- We depend on Java frontend code setting an env var (MOZ_LINKER_EXTRACT=1), based on a build flag (see GeckoLoader.java)
- which causes our c++ code to follow the wrong path when loading shared libs, since the env var isn't set (see ElfLoader.cpp)
- Which is indirectly caused by different build variables being set while building artifacts, and building locally (MOZ_LINKER_EXTRACT is only set for $COMPILE_ENVIRONMENT builds, which isn't the case for artifact builds, even though the artifacts that are being used might have been built that way.

Would it be better to instead move the build-flag dependent check directly into our C++ code, so that artifacts consistently load the correct library type based on whatever build config was used, without the frontend build having to care about how exactly libraries are loaded?

I.e. in ElfLoader.cpp, could we "#ifdef MOZ_LINKER_EXTRACT" instead of having a conditional on "getenv("MOZ_LINKER_EXTRACT");"?
See above comment.
Flags: needinfo?(mh+mozilla)
I'd rather not change the linker code for that. Just unconditionally putenv on the java end.
Flags: needinfo?(mh+mozilla)
Comment hidden (mozreview-request)
Assignee: nobody → ahunt
Status: NEW → ASSIGNED
Iteration: --- → 1.8
Priority: -- → P1
Whiteboard: [MobileAS]

Comment 8

3 years ago
mozreview-review
Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

https://reviewboard.mozilla.org/r/90028/#review90388

With this the MOZ_LINKER_EXTRACT build flag becomes redundant, I think it would be better to make MOZ_LINKER_EXTRACT=1 the default for Android in old-configure.in instead.
Attachment #8806650 - Flags: review?(mh+mozilla)

Comment 9

3 years ago
mozreview-review
Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

https://reviewboard.mozilla.org/r/90028/#review91166

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/mozglue/GeckoLoader.java:194
(Diff revision 2)
>          String forced_ondemand = System.getenv("MOZ_LINKER_ONDEMAND");
>          if (forced_ondemand == null) {
>              if ("HTC".equals(android.os.Build.MANUFACTURER) &&
>                  "HTC Vision".equals(android.os.Build.MODEL)) {
>                  putenv("MOZ_LINKER_ONDEMAND=0");
>              }
>          }

BTW, this can go away.

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/mozglue/GeckoLoader.java:202
(Diff revision 2)
> -        if (AppConstants.MOZ_LINKER_EXTRACT) {
> +        // Artifact builds might not have MOZ_LINKER_EXTRACT set due to their separate build config,
> +        // however the artifacts that they use will have been built using MOZ_LINKER_EXTRACT,
> +        // hence we need to pass MOZ_LINKER_EXTRACT to the linker regardless of actual build config.

I don't think this comment needs to mention artifact builds. The main point is that what Firefox for Android wants now is that the linker always extracts files, and that's how this is made to happen because, as of now, it's not what the linker does by default.

::: mobile/android/geckoview/src/main/java/org/mozilla/gecko/mozglue/GeckoLoader.java:202
(Diff revision 2)
>                  "HTC Vision".equals(android.os.Build.MODEL)) {
>                  putenv("MOZ_LINKER_ONDEMAND=0");
>              }
>          }
>  
> -        if (AppConstants.MOZ_LINKER_EXTRACT) {
> +        // Artifact builds might not have MOZ_LINKER_EXTRACT set due to their separate build config,

You can also remove the constant.
Attachment #8806650 - Flags: review?(mh+mozilla) → review+

Comment 10

3 years ago
mozreview-review
Comment on attachment 8806650 [details]
Bug 1314546 - Always putenv(MOZ_LINKER_EXTRACT) for Android

https://reviewboard.mozilla.org/r/90028/#review91280
Attachment #8806650 - Flags: review?(esawin) → review+
Comment hidden (mozreview-request)

Comment 12

3 years ago
Pushed by ahunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/720cb3b359c9
Always putenv(MOZ_LINKER_EXTRACT) for Android r=esawin,glandium

Comment 13

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/720cb3b359c9
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 52
You need to log in before you can comment on or make changes to this bug.