1844964 - Crash in [@ java.lang.NoClassDefFoundError: at kotlinx.coroutines.channels.ChannelKt.Channel$default(Channel.kt:44)] kotlinx.coroutines.channels.BufferedChannel

Status: NEW

(Fenix :: General, defect, P3)

Description

[Chris Peterson [:cpeterson]]

Crash report: https://crash-stats.mozilla.org/report/index/c50fff7b-2f92-4f30-9a92-bf13f0230712

I think this crash is either a regression in 115 or the signature changed in 115, to:

[@ java.lang.NoClassDefFoundError: at kotlinx.coroutines.channels.ChannelKt.Channel$default(Channel.kt:48) ] and exception java.lang.NoClassDefFoundError: <clinit> failed for class kotlinx.coroutines.channels.AbstractChannel; see exception in other thread.

from:

[@ java.lang.NoClassDefFoundError: at kotlinx.coroutines.channels.ChannelKt.Channel$default(Channel.kt:48) ] and exception java.lang.NoClassDefFoundError: kotlinx.coroutines.channels.BufferedChannel

Though both signatures appear in 115. The second last appears in 115.

Java stack trace:

java.lang.NoClassDefFoundError: kotlinx.coroutines.channels.BufferedChannel
	at kotlinx.coroutines.channels.ChannelKt.Channel$default(Channel.kt:44)
	at androidx.datastore.core.SimpleActor.<init>(SimpleActor.kt:18)
	at androidx.datastore.core.SingleProcessDataStore.<init>(SingleProcessDataStore.kt:68)
	at org.mozilla.fenix.components.Components$appStore$2.invoke(Components.kt:302)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:21)
	at org.mozilla.fenix.components.Components.getAppStore(Components.kt:3)
	at org.mozilla.fenix.components.Components$useCases$2.invoke(Components.kt:59)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:21)
	at org.mozilla.fenix.components.Components.getUseCases(Components.kt:3)
	at org.mozilla.fenix.FenixApplication.initialize(FenixApplication.kt:701)
	at org.mozilla.fenix.FenixApplication.onCreate(FenixApplication.kt:4)
	at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1020)
	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5109)
	at android.app.ActivityThread.access$1600(ActivityThread.java:177)
	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1509)
	at android.os.Handler.dispatchMessage(Handler.java:102)
	at android.os.Looper.loop(Looper.java:145)
	at android.app.ActivityThread.main(ActivityThread.java:5938)
	at java.lang.reflect.Method.invoke(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:372)
	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1400)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1195)

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 AArch64 and ARM crashes on beta

For more information, please visit BugBot documentation.

Comment 3

[Chris Peterson [:cpeterson]]

I think bug 1846566 is a duplicate of this bug. In that bug, Aryx says:

Startup crash with Android 5.0 on Samsung devices (only?), new with Firefox for Android 116.0, 17 crash reports from 11 installations at this moment.

Crash report: https://crash-stats.mozilla.org/report/index/5bcfbd2c-dcbf-419f-9c17-079cc0230801

Java stack trace:

java.lang.NoClassDefFoundError: kotlinx.coroutines.channels.BufferedChannelKt
	at kotlinx.coroutines.channels.BufferedChannel.<init>(BufferedChannel.kt:15)
	at kotlinx.coroutines.channels.ChannelKt.Channel$default(Channel.kt:46)
	at androidx.datastore.core.SimpleActor.<init>(SimpleActor.kt:18)
	at androidx.datastore.core.SingleProcessDataStore.<init>(SingleProcessDataStore.kt:68)
	at org.mozilla.fenix.components.Components$appStore$2.invoke(Components.kt:302)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:21)
	at org.mozilla.fenix.components.Components.getAppStore(Components.kt:3)
	at org.mozilla.fenix.components.Components$useCases$2.invoke(Components.kt:59)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:21)
	at org.mozilla.fenix.components.Components.getUseCases(Components.kt:3)
	at org.mozilla.fenix.FenixApplication.initialize(FenixApplication.kt:703)
	at org.mozilla.fenix.FenixApplication.onCreate(FenixApplication.kt:4)
	at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1020)
	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5245)
	at android.app.ActivityThread.access$1600(ActivityThread.java:181)
	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1537)
	at android.os.Handler.dispatchMessage(Handler.java:102)
	at android.os.Looper.loop(Looper.java:145)
	at android.app.ActivityThread.main(ActivityThread.java:6134)
	at java.lang.reflect.Method.invoke(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:372)
	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194)

Comment 4

[Dianna Smith [:diannaS]]

Is there anything we can do here atleast for 117? Seems to be pretty high volume for 116

Comment 5

[Chris Peterson [:cpeterson]]

[@ java.lang.NoClassDefFoundError: at kotlinx.coroutines.channels.BufferedChannel.<init>(BufferedChannel.kt:15)] is a slightly different crash signature for the same crash stack trace:

Crash report: https://crash-stats.mozilla.org/report/index/d828696f-9180-4905-8129-89d1c0230815

Java stack trace:

java.lang.NoClassDefFoundError
	at kotlinx.coroutines.channels.BufferedChannel.<init>(BufferedChannel.kt:15)
	at kotlinx.coroutines.channels.ChannelKt.Channel$default(Channel.kt:46)
	at androidx.datastore.core.SimpleActor.<init>(SimpleActor.kt:18)
	at androidx.datastore.core.SingleProcessDataStore.<init>(SingleProcessDataStore.kt:68)
	at org.mozilla.fenix.components.Components$appStore$2.invoke(Components.kt:302)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:21)
	at org.mozilla.fenix.components.Components.getAppStore(Components.kt:3)
	at org.mozilla.fenix.components.Components$useCases$2.invoke(Components.kt:59)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:21)
	at org.mozilla.fenix.components.Components.getUseCases(Components.kt:3)
	at org.mozilla.fenix.FenixApplication.initialize(FenixApplication.kt:703)
	at org.mozilla.fenix.FenixApplication.onCreate(FenixApplication.kt:4)
	at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1020)
	at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5237)
	at android.app.ActivityThread.access$1600(ActivityThread.java:181)
	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1536)
	at android.os.Handler.dispatchMessage(Handler.java:102)
	at android.os.Looper.loop(Looper.java:145)
	at android.app.ActivityThread.main(ActivityThread.java:6126)
	at java.lang.reflect.Method.invoke(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:372)
	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194)

Comment 6

[Chris Peterson [:cpeterson]]

99.98% of these crash reports are from Android 5.0 (API 21).

Comment 7

[Chris Peterson [:cpeterson]]

I suspect this crash is a regression bug 1838274, which updated the Kotlin coroutines library from version 1.6.4 to 1.7.1 in Fx116.

Bug 1841081 updated from 1.7.1 to 1.7.2 in Fx117, but we're still crashing.

Comment 8

[Jonathan Almeida [:jonalmeida]]

Bug 1838274 should not be the regression. The kotlin coroutine library update in GeckoView is not used in production code.

Bug 1826590 does seem like the most likely cause and it did land in 116 as well. I will try to spend a few cycles to see if there is some change in the release notes that we missed.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1826590

Comment 10

[Jonathan Almeida [:jonalmeida]]

All of these crashes are happening on API 21 devices from the linked crash reports.

They are all originating with the same stack trace within the Datastore library that uses a much earlier version of kotlin coroutines (1.5.0) that we no longer use (currently on 1.7.2), so we end up upgrading the library to our version. The alpha version of the data store library has updated to the latest coroutine library, so I suspect that might fix it (eventually) but we can't rely on this alpha library right now.

This doesn't answer yet why the BufferedChannelKt is missing when the BufferedChannel class exists and is part of the initialization stack. csadilek and I went through some of the most common theories and it seems more like a bug in the coroutine library than ours.

Filed: https://github.com/Kotlin/kotlinx.coroutines/issues/3882

Comment 11

[Jonathan Almeida [:jonalmeida]]

From the feedback in the coroutine bug and more discussions offline, it doesn't seem like there is anything else actionable we can do right now.

• There is nothing special the coroutine library does with the classloader.
• The datastore library doesn't seem to have any special code path for API 21. There are only two instances at this point:
  1. An IPC logger (IpcLogger.kt), that is used for multi-process datastores (which we do not use).
  2. SharedPreferencesMigration.kt which we also have not exercised in our code base.
• This doc line, "This should only be used from a single application in a single classloader in a single process.", that made us consider if we are somehow initializing data store, however all code paths to our usages (Pocket and Cookie Banner storage) are always done in the main process.

This leaves only something specific to API 21 platform code that has bugs with coroutine 1.7.0, so we can file a google ticket for that (with low expectations of a fix).

TL;DR: we need to wait to update datastore to version 1.1 when it is release-ready as it includes the coroutine 1.7.0 library upgrade.

Comment 12

[Emilio Cobos Álvarez (:emilio)]

TL;DR: we need to wait to update datastore to version 1.1 when it is release-ready as it includes the coroutine 1.7.0 library upgrade.

Do you know what is the ETA for the release of datastore?

Comment 13

[Jonathan Almeida [:jonalmeida]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #12)

TL;DR: we need to wait to update datastore to version 1.1 when it is release-ready as it includes the coroutine 1.7.0 library upgrade.

Do you know what is the ETA for the release of datastore?

No, unfortunately.

Version 1.0.0 was released in Aug 2021 and the the 1.1.0 alpha05 was released in Sept 2023. 1.0.0 previously had a release cadence of once each month until it's final release. There doesn't seem to be a fixed time between any of the releases any more from what I can see any more.

Comment 14

[Donal Meehan [:dmeehan]]

Version 1.1.0 is still in alpha
Version 1.1.0-alpha06 released on November 1, 2023

Comment 15

[Donal Meehan [:dmeehan]]

No updates since comment 14

Comment 16

[Donal Meehan [:dmeehan]]

Version 1.1.0 is still in alpha
Version 1.1.0-alpha07 released on November 29, 2023