1989864 - Crash in [@ mozilla.appservices.remotesettings.InternalException: at mozilla.appservices.remotesettings.Remote_settingsKt.uniffiCheckCallStatus(remote_settings.kt:40)]

Status: NEW

(Firefox for Android :: General, defect)

Description

[Denis Müller [:Webworkr]]

Crash report: https://crash-stats.mozilla.org/report/index/fc73f56e-b47d-4d76-a53c-f128a0250922

Top 10 frames:

0  mozilla.appservices.remotesettings.Remote_settingsKt  uniffiCheckCallStatus  remote_settings.kt:40
1  mozilla.appservices.remotesettings.Remote_settingsKt  access$uniffiCheckCallStatus  remote_settings.kt:1
2  mozilla.appservices.remotesettings.RemoteSettings  getRecords  remote_settings.kt:60
3  mozilla.components.support.remotesettings.RemoteSettingsClient$fetch$2  invokeSuspend  RemoteSettingsClient.kt:16
4  mozilla.components.support.remotesettings.RemoteSettingsClient$fetch$2  invoke  RemoteSettingsClient.kt:13
5  kotlinx.coroutines.intrinsics.UndispatchedKt  startUndspatched  Undispatched.kt:19
6  kotlinx.coroutines.BuildersKt  withContext
7  mozilla.components.feature.search.telemetry.SerpTelemetryRepository  fetchRemoteResponse$feature_search_release  SerpTelemetryRepository.kt:68
8  mozilla.components.feature.search.telemetry.SerpTelemetryRepository  fetchRemoteResponse-S8_Dj7E$feature_search_release  SerpTelemetryRepository.kt:70
9  mozilla.components.feature.search.telemetry.SerpTelemetryRepository  updateProviderList  SerpTelemetryRepository.kt:179

Comment 1

[Denis Müller [:Webworkr]]

As far as I remember, the browser was in private mode. However, no tabs were open at the time of the crash.

Comment 2

[Denis Müller [:Webworkr]]

Restarting after the crash was not possible and led to the following crash report:
https://crash-stats.mozilla.org/report/index/27989909-cd9e-44e0-9f68-0a1e30250922 (referenced bug reports: bug 1646397, bug 1927161).

Comment 3

[Denis Müller [:Webworkr]]

This in turn led to various futile attempts to restart the browser, which initially could not be remedied even by forcefully closing the browser:
bug 1989865

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The bug has a crash signature, thus the bug will be considered confirmed.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 AArch64 and ARM crashes on nightly

For more information, please visit BugBot documentation.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

These crashes are still happening after bug 1989865 landed.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Various stacks here, but all of them are showing the same exception: InternalException: null RustBuffer had non-zero capacity

mozilla.appservices.remotesettings.InternalException: null RustBuffer had non-zero capacity
	at mozilla.appservices.remotesettings.Remote_settingsKt.uniffiCheckCallStatus(remote_settings.kt:40)
	at mozilla.appservices.remotesettings.Remote_settingsKt.access$uniffiCheckCallStatus(remote_settings.kt:1)
	at mozilla.appservices.remotesettings.RemoteSettings.getRecords(remote_settings.kt:60)
	at mozilla.components.support.remotesettings.RemoteSettingsClient$fetch$2.invokeSuspend(RemoteSettingsClient.kt:16)
	at mozilla.components.support.remotesettings.RemoteSettingsClient$fetch$2.invoke(RemoteSettingsClient.kt:13)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:19)
	at kotlinx.coroutines.BuildersKt.withContext(Unknown Source:78)
	at mozilla.components.feature.search.telemetry.SerpTelemetryRepository.fetchRemoteResponse$feature_search_release(SerpTelemetryRepository.kt:68)
	at mozilla.components.feature.search.telemetry.SerpTelemetryRepository.fetchRemoteResponse-S8_Dj7E$feature_search_release(SerpTelemetryRepository.kt:70)
	at mozilla.components.feature.search.telemetry.SerpTelemetryRepository.updateProviderList(SerpTelemetryRepository.kt:179)
	at org.mozilla.fenix.components.Core$store$2$1$1$providerList$1.invokeSuspend(Core.kt:62)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:9)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:122)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:4)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:3)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:94)
	Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: null

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox145 (nightly). We have limited time to fix this, the soft freeze is in 3 days. However, the bug still isn't assigned.

:boek, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 10

[Mark Hammond [:markh] [:mhammond]]

https://crash-stats.mozilla.org/report/index/efae5968-8ad2-42c7-9dd4-e35690251013 has mozilla.appservices.remotesettings.InternalException: RustBuffer length exceeds capacity as the message with the same stack as above.

This is a rust panic - why is it crashing? Is it just an unhandled exception?

Comment 11

[Ben Dean-Kawamura [:bdk]]

Maybe I'm becoming a JNA conspiracy theorist, but I feel like the underlying cause is JNA bugs:

• We only see them in Kotlin
• capacity >= length is a pretty basic invariant that I doubt we're breaking
• The checks I've added to try to catch the null RustBuffer crash don't seem to be catching anything.
• In general, we don't have any theories on why our code is wrong.

This makes me think it's something to do with how JNA is reading/writing the structs. Alternatively, it could be how we're using JNA, like maybe we're missing a read call somewhere.

I'd really like to move away from JNA -- especially having JNA deal with passing structs across the FFI. This is definitely not a short-term fix. I'm hoping to prioritize this, but we're still talking about the end of the year at the earliest.

In the meantime, What about adding more catch blocks in the Kotlin code like Mark suggests? I don't think that fixes the root cause, but maybe it mitigates the worst? One issue might be getting a reasonable return value in those cases -- what do you do if the constructor fails? Maybe we try making the same call again and see if it works?

Comment 12

[Ben Dean-Kawamura [:bdk]]

Mark and I talked about this and we think there's some decent short-term solutions for this. I made https://bugzilla.mozilla.org/show_bug.cgi?id=1994283 for that work.

Comment 13

[Andrew McCreight [:mccr8]]

I'm not sure why this crash signature has like a UUID in it but I assume this is the same basic issue: [@ mozilla.appservices.remotesettings.InternalException: at mozilla.appservices.remotesettings.Remote_settingsKt.uniffiCheckCallStatus(r8-map-id-ea7ddee2f424ddb43c5acaae0ed48e74c7a9d46d23584be9afa21737e43e6228:40) ]

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

This is a reminder regarding comment #9!

The bug is marked as tracked for firefox145 (beta). We have limited time to fix this, the soft freeze is in 14 days. However, the bug still isn't assigned.