1791842 - Android crash in [@ java.lang.UnsatisfiedLinkError: at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath]

Status: REOPENED

(Firefox for Android :: General, defect, P5)

Description

[Ryan VanderMeulen [:RyanVM]]

This appears to be a longstanding mobile crash, though the volume seems to be a lot higher in 105 so far. Filing this under Glean for now given the stack, but feel free to move it elsewhere as needed.

Crash report: https://crash-stats.mozilla.org/report/index/7e6d033e-fe27-45ee-898b-1a70b0220921

Java stack trace:

java.lang.UnsatisfiedLinkError
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:14)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:35)
	at com.sun.jna.Native.<clinit>(Native.java:12)
	at com.sun.jna.Native.getStructureAlignment(Native.java:1)
	at com.sun.jna.Structure.setAlignType(Structure.java:2)
	at com.sun.jna.Structure.<init>(Structure.java:12)
	at com.sun.jna.Structure.<init>(Structure.java:6)
	at com.sun.jna.Structure.<init>(Structure.java:3)
	at com.sun.jna.Structure.<init>(Structure.java:1)
	at mozilla.telemetry.glean.internal.RustCallStatus.<init>(glean.kt:1)
	at mozilla.telemetry.glean.internal.GleanKt.gleanEnableLogging(glean.kt:2)
	at mozilla.telemetry.glean.GleanInternalAPI.<init>(Glean.kt:4)
	at mozilla.telemetry.glean.Glean.<init>(Glean.kt:1)
	at mozilla.telemetry.glean.Glean.<clinit>(Glean.kt)
	at org.mozilla.fenix.FenixApplication.onCreate(FenixApplication.kt:25)
	at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1024)
	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5474)
	at android.app.ActivityThread.-wrap2(ActivityThread.java)
	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1550)
	at android.os.Handler.dispatchMessage(Handler.java:102)
	at android.os.Looper.loop(Looper.java:154)
	at android.app.ActivityThread.main(ActivityThread.java:6190)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:892)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:782)

Comment 1

[Kevin Brosnan [Ex-Mozilla]]

This list is overwhelmingly emulated devices.

java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/android-x86/libjnidispatch.so) not found in resource path (.)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:14)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:35)
	at com.sun.jna.Native.<clinit>(Native.java:12)
	at com.sun.jna.Native.getStructureAlignment(Native.java:1)
	at com.sun.jna.Structure.setAlignType(Structure.java:2)
	at com.sun.jna.Structure.<init>(Structure.java:12)
	at com.sun.jna.Structure.<init>(Structure.java:6)
	at com.sun.jna.Structure.<init>(Structure.java:3)
	at com.sun.jna.Structure.<init>(Structure.java:1)
	at mozilla.telemetry.glean.internal.RustCallStatus.<init>(glean.kt:1)
	at mozilla.telemetry.glean.internal.GleanKt.gleanEnableLogging(glean.kt:2)
	at mozilla.telemetry.glean.GleanInternalAPI.<init>(Glean.kt:4)
	at mozilla.telemetry.glean.Glean.<init>(Glean.kt:1)
	at mozilla.telemetry.glean.Glean.<clinit>(Glean.kt)
	at org.mozilla.fenix.FenixApplication.onCreate(FenixApplication.kt:25)

arm64-v8a example

java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/android-aarch64/libjnidispatch.so) not found in resource path (.)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:14)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:35)
	at com.sun.jna.Native.<clinit>(Native.java:12)
	at com.sun.jna.Native.getStructureAlignment(Native.java:1)
	at com.sun.jna.Structure.setAlignType(Structure.java:2)
	at com.sun.jna.Structure.<init>(Structure.java:12)
	at com.sun.jna.Structure.<init>(Structure.java:6)
	at com.sun.jna.Structure.<init>(Structure.java:3)
	at com.sun.jna.Structure.<init>(Structure.java:1)
	at mozilla.telemetry.glean.internal.RustCallStatus.<init>(glean.kt:1)
	at mozilla.telemetry.glean.internal.EventMetric.<init>(glean.kt:3)
	at mozilla.telemetry.glean.private.EventMetricType.<init>(EventMetricType.kt:3)
	at org.mozilla.experiments.nimbus.GleanMetrics.NimbusHealth$sdkUnavailableForFeature$2.invoke(NimbusHealth.kt:7)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:5)
	at org.mozilla.experiments.nimbus.internal.FeatureHolder$value$1.invoke(FeatureHolder.kt:8)
	at org.mozilla.experiments.nimbus.internal.FeatureHolder.runBlock(FeatureHolder.kt:2)
	at org.mozilla.experiments.nimbus.internal.FeatureHolder.value$default(FeatureHolder.kt:2)
	at org.mozilla.fenix.utils.Settings.getMr2022Sections(Settings.kt:3)
	at org.mozilla.fenix.utils.Settings.<init>(Settings.kt:183)
	at org.mozilla.fenix.components.Components$settings$2.invoke(Components.kt:3)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:5)
	at org.mozilla.fenix.components.Components.getSettings(Components.kt:1)

Some sort of running an for a specific architecture APK on the wrong CPU?

Comment 2

[Travis Long [:travis_]]

I'm moving this to the Glean: SDK component and taking a look at this today. Thanks Kevin, for looking at this by platform. I think that points to what may be going on here. I'll update this bug with what I find.

Comment 3

[Travis Long [:travis_]]

Okay, I believe the problem here is that Glean doesn't bundle native x86 stuff into the binary that is used with Fenix. That's why there is a glean and a gleanForUnitTests published by Glean, the x86 binaries are in gleanForUnitTests.

I don't know that this is a problem we should address:

• Less than 1% of all clients reported in telemetry are x86_64 or x86 architectures
• According to Sentry:
  • 75% of these errors originate from a locale of zh_CN
  • 84% indicate a rooted OS
  • 64% have an installer store of com.android.flysilkworm and not the Android Play Store

The fact that this impacts so few, and those few appear to be suspicious in nature leads me to believe that this crash is mostly just noise and not indicative of a problem we can easily address. Closing this as WONTFIX

Comment 4

[Ryan VanderMeulen [:RyanVM]]

(In reply to Travis Long [:travis_] from comment #3)

Okay, I believe the problem here is that Glean doesn't bundle native x86 stuff into the binary that is used with Fenix.

Why is that? We ship native binaries across all 4 architectures for libxul and A-S AFAIK.

Comment 5

[Travis Long [:travis_]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #4)

(In reply to Travis Long [:travis_] from comment #3)

Okay, I believe the problem here is that Glean doesn't bundle native x86 stuff into the binary that is used with Fenix.

Why is that? We ship native binaries across all 4 architectures for libxul and A-S AFAIK.

It's possible we just aren't packaging the right libjnidispatch.so with the Glean that's shipping with libxul. I'm less familiar with how Glean gets built in m-c for geckoview. perhaps Jan-Erik may have some idea. I can reopen this if there is something we can do to fix this.

Comment 6

[Chris Peterson [:cpeterson]]

Is this bug related to the Glean UnsatisfiedLinkError regression (bug 1792799) in Fenix 94.3.0?

:bdk just rolled back an Android NDK upgrade for that bug in https://github.com/mozilla/application-services/issues/5165.

Comment 7

[Alessio Placitelli [:Dexter]]

Looks like A-S fixed this by downgrading from NDK 25 back to NDK 21. Here's the bug from NDK 25: https://github.com/android/ndk/issues/1614

Glean is using NDK 25 since version 51.3.0. So this could be a likely cause.

Comment 8

[Alessio Placitelli [:Dexter]]

Let's reopen this, as from the other A-S bugs this seems to happen on a few devices outside of the emulator, so we need to decide how to handle this.

Comment 9

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

I'm pretty sure we ship x86(_64) stuff for Android, so the NDK issue seems a likely cause.
I'm preparing a revert of that upgrade and will test it locally.

Comment 10

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

I wasn't able to reproduce the crash in the emulator (neither on x86 nor x86_64). Regardless a downgrade is easy to do so we can roll that out and see if that helps.

Comment 11

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

Attachment: pr 2227: Bug 1791842 - Revert "Kotlin: Upgrade to NDK r25"

Comment 12

[Ryan VanderMeulen [:RyanVM]]

I thought about that possibility too, but this spike started in Fx105 when we were still on Glean 51.1.0 AFAICT
https://github.com/mozilla-mobile/android-components/blob/releases/105.0/buildSrc/src/main/java/Dependencies.kt#L38

Comment 13

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #12)

I thought about that possibility too, but this spike started in Fx105 when we were still on Glean 51.1.0 AFAICT
https://github.com/mozilla-mobile/android-components/blob/releases/105.0/buildSrc/src/main/java/Dependencies.kt#L38

Hm, that means we need another look here.

Comment 14

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

Reverting our NDK upgrade might have been premature indeed, given we don't see the same issues a-s saw.

Our official builds (those provided on github releases) come with the right libraries:

~$ unzip -l fenix-106.1.0-x86_64.apk | rg 'jni|xul'
   101720  01-01-1981 01:01   lib/x86_64/libjnidispatch.so
132371000  01-01-1981 01:01   lib/x86_64/libxul.so

~$ unzip -l fenix-105.2.0-x86.apk | rg 'jni|xul'
   103428  01-01-1981 01:01   lib/x86/libjnidispatch.so
146903360  01-01-1981 01:01   lib/x86/libxul.so

Running Android 7 (what most crashes are from) in the emulator (x86), I can see libjnidispatch.so is in the expected location.

generic_x86:/ # ls -l /data/app/org.mozilla.firefox-1/lib/x86/libjnidispatch.so
-rwxr-xr-x 1 system system 103428 1981-01-01 01:01 /data/app/org.mozilla.firefox-1/lib/x86/libjnidispatch.so

Fenix runs without issues.
If I remove that file, I get the expected crash.

         AndroidRuntime  E  FATAL EXCEPTION: main
                         E  Process: org.mozilla.firefox, PID: 4912
                         E  java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/android-x86/libjnidispatch.so) not found in resource path (.)
                         E      at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:14)
                         E      at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:35)

For proper x86 devices our own build seem to be fine.

---

Looking at the Sentry report (the one travis also looked at):

Device reported as SKW-A0, which seems to be the Xiaomi Black Shark 2 SKW-A0.
That one has a Snapdragon 855, which has a 64-bit ARM processor.
It reports Architectures: [x86, armeabi-v7a, armeabi].

How is that possible? Why is it reporting x86?
The Fenix ARM build of course doesn't ship the x86 libraries, so if the system tries to load them it won't find them. Not much we can do about that I guess.
Happy to hear from someone with more Android experience about what's going on.

(Why this only happened with 105? I don't know. As Travis correctly said the majority of these crash reports on Sentry seem to be from a non-google-appstore. Did they only now start shipping builds and if so, do they build on their own or is that our builds?)

Comment 15

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

Attachment: pr 2228: Bug 1791842 - Revert "Revert "Kotlin: Upgrade to NDK r25""

This is the revert of the revert. Again. I'd rather have us ship with the latest NDK, but I'm holding back until we get a final say about it in the bug.

Comment 16

[Donal Meehan [:dmeehan]]

:janerick has there been any progress on this in the last 2 weeks
Wondering if we can expect anything for 108?

Comment 17

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

No. Didn't get a chance for followups.
Tomorrow I'll try to find some Android engineers who can help.

Comment 18

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

Can you help out here or let me know some Android engineer who could?
Specifically the architecture mismatches I documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1791842#c14

Comment 19

[Chris Peterson [:cpeterson]]

(In reply to Jan-Erik Rediger [:janerik] from comment #18)

Can you help out here or let me know some Android engineer who could?
Specifically the architecture mismatches I documented in https://bugzilla.mozilla.org/show_bug.cgi?id=1791842#c14

:csadilek says you can ask him any questions you might have, but he doesn't have cycles to help debug it himself. He suggested we add more logging to UniFFI.

OTOH, we had fewer than 600 reports for this crash signature submitted from Fenix 106 and 80% of those are x86 devices or emulators. Maybe we should lower this bug's priority to P5 (or WONTFIX it again) and not invest any more time debugging it.

Comment 20

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

(In reply to Chris Peterson [:cpeterson] from comment #19)

OTOH, we had fewer than 600 reports for this crash signature submitted from Fenix 106 and 80% of those are x86 devices or emulators. Maybe we should lower this bug's priority to P5 (or WONTFIX it again) and not invest any more time debugging it.

That works for me.

As one last resort though:
:csadilek, see full explanation at https://bugzilla.mozilla.org/show_bug.cgi?id=1791842#c14

Why would a device with an ARM processor report its architecture as x86?
(It's fine if you don't have an answer to that, as per suggestion above I'll downgrade priority of this bug in a couple of days then)

Comment 21

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

Ah, this was asked in slack as well already. No need for further insight then, guess :csadilek did say all he knew.

Comment 22

[twhite]

Re-opening this bug as we are seeing spikes in UnUnsatisfiedLinkError again around Dec 22nd.

• Sentry overview
• Sentry crash
• Socorro report

Comment 23

[Chris Peterson [:cpeterson]]

Looks like this new spike is a regression in Fx 121 (release date: 2023-12-19). We've received 551 crash reports from Fx 120, but 2625 from Fx 121. There's no correlation with OS version, device model, CPU architecture, or user locale.

https://crash-stats.mozilla.org/search/?signature=~UnsatisfiedLinkError%3A%20at%20com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath&product=Fenix&product=Focus&date=%3E%3D2023-07-05T23%3A13%3A00.000Z&date=%3C2024-01-05T23%3A13%3A00.000Z&_facets=signature&_facets=version&_facets=major_version&_facets=cpu_arch&_facets=android_version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-major_version

Comment 24

[Jan-Erik Rediger [:janerik] (Away 2026-04-29 - 2026-05-01]

I clicked through the list of recent crash reports (and the linked sentry report) and they all result from calling into app-services here.

This is different from the initial reported error that went through Glean code.
Looping in Ben for some help

Comment 25

[Donal Meehan [:dmeehan]]

:janerik/:cpeterson looks like we also have Bug 1792799 tracking the same crash? Could you confirm, and close one as a duplicate?

Comment 26

[Ben Dean-Kawamura [:bdk]]

I took another pass at this one today and still have no clue. I doubt it's related to either the app-services logging code or the Glean logging code, those are probably just the first things to get executed. It's possible that it's related to how app-services is packaging everything, but I can't figure out how. I looked through the v121 changelog and didn't see anything that stood out at all.

Searching for the error itself didn't turn up much other than the ReLinker library, which claims to fix intermittent link errors. However, it says that Android Marshmallow (API 23) fixed the underlying issue. I believe the minimum SDK is below 23, but most of the reports are above 23, so I doubt this will fix things.

Comment 27

[Ben Dean-Kawamura [:bdk]]

I've been reading this post from the ReLinker people over and over and they mention that they noticed people installing manually APKs in the wild, but with the wrong arch. This seems like one potential source of these crashes, not sure how wide-spread it is. Does this seem possible?

Would it make sense to do some sort-of check startup for this? I think that test would be to try to load the libjnidispatch and maybe some other native libraries, if that failed then crash with a more helpful message. For example, a list of all files inside the APK (the post says you can get that from Context.getApplicationInfo().sourceDir;. Also, what installed the APK, which the post says can be found withContext.getPackageManager().getInstallerPackageName().

Comment 28

[Chris Peterson [:cpeterson]]

(In reply to Donal Meehan [:dmeehan] from comment #25)

:janerik/:cpeterson looks like we also have Bug 1792799 tracking the same crash? Could you confirm, and close one as a duplicate?

I'll resolve bug 1792799 as a dupe of this bug because this bug has more details about the problem.

Comment 30

[BugBot [:suhaib / :marco/ :calixte]]

Copying crash signatures from duplicate bugs.

Comment 31

[Ben Dean-Kawamura [:bdk]]

I went ahead and implemented the check I described above in this PR: https://github.com/mozilla-mobile/firefox-android/pull/5204/files. It only runs when FF is about to crash because of an UnsatisfiedLinkError. I'd love review on this, but I'm not really sure who to ask for it.

Comment 32

[BugBot [:suhaib / :marco/ :calixte]]

Copying crash signatures from duplicate bugs.

Comment 33

[Ben Dean-Kawamura [:bdk]]

Looking through the socorro reports, it looks like this happens on the first run after install. Basically all crashes on the first page were from the same day as install, almost all of them < 5 minutes after install. The only exceptions were the 2 crashes from v116 installs, I think they're just running into the older issue that was already fixed.

Comment 34

[Donal Meehan [:dmeehan]]

:bdk did the investigation produce anything to mitigate this?
Due to the crash spike the severity should by increased?
Did we find out what caused the spike in 121? Something new in that version that explains the increase?

Comment 35

[Ben Dean-Kawamura [:bdk]]

Thanks for circling back on this one. There's been a few sentry events back for this issue and I'd love you're help deciphering them.

• All of the events list no files at all in the lib dirs -- both for the the APK and install dir. This seems like it's the culprit to me, although it seems pretty unusual to have no files at all. Maybe there's a difference in permissions or something on a real phone vs my emulator and that's causing the code to incorrectly report 0 files?
• The installer package names that I saw were: com.apkpure.aegon, com.miui.packageinstaller, com.android.secex, com.android.packageinstaller. The first 2 definitely seem like 3rd-party. The last 2 seem vaguely official, but shouldn't it be com.google.android? I'm out of my depth here.

I think there's decent evidence that these are invalid APKs coming from 3rd-party installers, but I don't have enough Android expertise to say that with certainty.

As to the spike, maybe the 3rd-parties decided to use the release to capture users or something? I'm really guessing here.

Comment 36

[Ben Dean-Kawamura [:bdk]]

Another theory is that the installs were always broken, but 121 made a code change that shifted where the breakage happened. So the spike isn't a new bug, it's just a new expression of an existing issue (or non-issue I guess).

Comment 37

[Donal Meehan [:dmeehan]]

:cpeterson is there anyone on the Android team that can assist with Comment 35?

Comment 38

[Chris Peterson [:cpeterson]]

(In reply to Donal Meehan [:dmeehan] from comment #37)

:cpeterson is there anyone on the Android team that can assist with Comment 35?

I'll follow up with some Android engineers.

Comment 39

[Chris Peterson [:cpeterson]]

(In reply to Ben Dean-Kawamura [:bdk] from comment #35)

• The installer package names that I saw were: com.apkpure.aegon, com.miui.packageinstaller, com.android.secex, com.android.packageinstaller. The first 2 definitely seem like 3rd-party. The last 2 seem vaguely official, but shouldn't it be com.google.android?

https://android.googlesource.com/platform/packages/apps/PackageInstaller/+/47fe118e0178e9d72c98073ff588ee5cf353258e/src/com/android/packageinstaller suggests at least com.android.packageinstaller is legit.

Comment 40

[Donal Meehan [:dmeehan]]

This is now the top crash in Firefox Android. Could the priority be raised from a P5?

Comment 41

[Jeff Boek [:boek]]

After some investigation with :bed we found that the source of this crash is coming from users installing the application through an alternative app store named Apkpure. We were able to reproduce the crash by doing a clean installation of Firefox on my S22 using that app store. Another piece of evidence is that other users are also reporting the same thing.

I think the timing of this and us moving to aab's is likely related, but I also don't want to rule out any shenanigans between us uploading an apk and it ending up there.

Comment 42

[Zac McKenney [:zmckenney]]

When we moved to AAB it appears it broke APKpure and other alternative install sites. Previously the APK's served from APKpure contained all the arch native libraries but after our move to AAB they are serving 3 different versions of the app which are broken on devices. They currently serve x86 @ mdpi, arm v7a @ xxhdpi, arm v7a @ xhdpi. I didn't look at the Xiaomi installer (com.miui.packageinstaller) but I am guessing they are also serving broken builds.

Comment 43

[Chris Peterson [:cpeterson]]

Looks like this crash spiked starting 2024-08-26, though that date doesn't align with any Fx release.

Also, I don't see a spike in crash ping telemetry on the Release channel: https://mozilla.cloud.looker.com/explore/fenix/crash?qid=CmH0dFqBBQrXiFK7xVAbnl&toggle=dat,pik,vis

Comment 44

[Mike Conley (:mconley) (:⚙️)]

Hey cpeterson,

So I think we've narrowed in on what's going on - this has to do with the teams serving up the wrong architectures in their third-party app stores. This problem isn't really tied to a release, and isn't something within our control. The advice moving forward with this one is to not set affected for any future releases. Thanks!

I've moved this to Fenix :: General. If the Android team wants to chase down these third-party app store maintainers to get this solved, we have a better bet of doing that in this component.

Comment 45

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criteria:

• Top 10 AArch64 and ARM crashes on beta
• Top 10 AArch64 and ARM crashes on release

:boek, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 46

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S4. However, the following bug duplicate has higher severity:

• Bug 1792799: S2

:boek, could you consider increasing the severity of this bug to S2?

For more information, please visit BugBot documentation.

Comment 47

[Chris Peterson [:cpeterson]]

(In reply to Mike Conley (:mconley) (:⚙️) from comment #44)

this has to do with the teams serving up the wrong architectures in their third-party app stores.

Mozilla teams or non-Mozilla people uploading Firefox to third-party app stores? Do you have an example crash report that indicates a different app store?

I've seen telemetry saying more than 50% of Firefox installs are from third-party app stores. Crashes like this might be a good reason for Mozilla to start uploading official builds to those app stores so users looking for Firefox get a better experience.

Comment 48

[Mike Conley (:mconley) (:⚙️)]

I think I'm going to redirect this to RyanVM, who knows more about this than I do (I was merely an avatar with a keyboard during that part of triage!)

Comment 49

[Ryan VanderMeulen [:RyanVM]]

See comments 41 and 42. I would defer any follow-up questions to the people on the Android team who investigated this previously.

Comment 50

[Chris Peterson [:cpeterson]]

I don't think we can resolve this bug as third-party app stores repackaging Firefox APKs incorrectly. IIRC, only about half of Firefox Android users installed from the Google Play Store, so it's not surprising we have many crashes from builds installed from other stores.

Here are recent Sentry crash reports from official Firefox 131 builds installed from the Google Play Store (installerStore com.android.vending and com.android.secex) being unable to load android-aarch64/libmegazord.so on arm64-v8a devices (Honor Magic5 Pro and Xiaomi Poco X3 Pro).

https://mozilla.sentry.io/issues/5368484135/
https://www.gsmarena.com/honor_magic5_pro-12148.php

https://mozilla.sentry.io/issues/5337826588/
https://www.gsmarena.com/xiaomi_poco_x3_pro-10802.php

Comment 51

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit BugBot documentation.

Comment 52

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Crash volume has jumped with the releases of Firefox for Android and Focus v144, almost exclusively for Android 9 (API 28).

Chris, Jeff, any insights what caused this?

Comment 53

[Donal Meehan [:dmeehan]]

Which signature is spiking? I'm not sure why I don't see it in Socorro?

Comment 54

[Ryan VanderMeulen [:RyanVM]]

Looks like [@ java.lang.UnsatisfiedLinkError: at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java)] is spiking on Focus specifically in AppServices init code. Jon, any thoughts? :-)

Example:
https://crash-stats.mozilla.org/report/index/7ef3d7b3-b4da-463b-8026-ec5930251023

Comment 55

[Ryan VanderMeulen [:RyanVM]]

Wait, these are all x86 crashes. Which 144+ isn't supposed to be supporting. HTH are these devices running it?

Comment 56

[Chris Peterson [:cpeterson]]

Looks like [@ java.lang.UnsatisfiedLinkError: at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java)] crash volume increased 2x in 137 and 5x in 138. Half of those crashes are from 32-bit ARM devices. The crashes don't look limited to any particular Android OS versions.

Comment 57

[Ryan VanderMeulen [:RyanVM]]

Fun story, Google Play thinks 144 still supports x86 due to the presence of a handful of native libraries from third-party dependencies which are present in the app bundle. I actually made us stop packaging those in bug 1990291 as a clean-up and we confirmed that 145+ only show as supporting 3 ABIs in the Play Console. I've requested uplift of that patch for 144.0.2, but it won't do anything to help the Chromebook users who already got updated to 144 :-\

Comment 58

[Ryan VanderMeulen [:RyanVM]]

I've added explicit device exclusions for Fenix/Focus/Klar to prevent it from being offered to x86 devices running Android 8+. It won't help the users who've already received the update, but it will prevent any others from getting it. Also, Play Store statistics suggest an upper bound of ~4000 affected installs for this issue.

Comment 59

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox144 (release). We have limited time to fix this, the soft freeze is in 13 days. However, the bug still isn't assigned and has low priority.

:boek, could you please find an assignee and increase the priority for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 60

[Jeff Boek [:boek]]

Ryan, is there anything else that can do with this?

Comment 61

[Ryan VanderMeulen [:RyanVM]]

(In reply to Jeff Boek [:boek] from comment #60)

Ryan, is there anything else that can do with this?

No, I think we've done what we can here. The volume is tailing off now and we're not getting any reports from 144.0.2 as expected.

Comment 62

[Donal Meehan [:dmeehan]]

Setting Fx144 to wontfix, but Bug 1990291 has fixed the recent spike in crashes re: Comment 52