1869225 - Crash in [@ mozilla.appservices.suggest.SuggestApiException$Other: at mozilla.appservices.suggest.FfiConverterTypeSuggestApiError.read(suggest.kt)] "SQL: database disk image is malformed"

Status: RESOLVED FIXED

(Firefox for Android :: Search, defect, P1)

Description

[Chris Peterson [:cpeterson]]

Crash report: https://crash-stats.mozilla.org/report/index/41cfba97-729b-4e1d-8524-b08a10231210

This crash looks like it might be a regression in 122.

Exception: mozilla.appservices.suggest.SuggestApiException$Other
Message: reason=Error executing SQL: database disk image is malformed

Frame   Module  Function  Source
0   mozilla.appservices.suggest.FfiConverterTypeSuggestApiError   read  suggest.kt:5
1   mozilla.appservices.suggest.FfiConverterTypeSuggestApiError   read  suggest.kt:1
2   mozilla.appservices.suggest.FfiConverter$DefaultImpls   liftFromRustBuffer  suggest.kt:13
3   mozilla.appservices.suggest.FfiConverterRustBuffer$DefaultImpls   liftFromRustBuffer  suggest.kt:6
4   mozilla.appservices.suggest.FfiConverterTypeSuggestApiError   liftFromRustBuffer  suggest.kt:2
5   mozilla.appservices.suggest.FfiConverterTypeSuggestApiError   liftFromRustBuffer  suggest.kt:1
6   mozilla.appservices.suggest.FfiConverterRustBuffer$DefaultImpls   lift  suggest.kt:7
7   mozilla.appservices.suggest.FfiConverterTypeSuggestApiError   lift  suggest.kt:3
8   mozilla.appservices.suggest.FfiConverterTypeSuggestApiError   lift  suggest.kt:2
10  mozilla.appservices.suggest.SuggestApiException$ErrorHandler  lift  suggest.kt:1
11  mozilla.appservices.suggest.SuggestKt   checkCallStatus   suggest.kt:75
12  mozilla.appservices.suggest.SuggestKt   access$checkCallStatus  suggest.kt:1
13  mozilla.appservices.suggest.SuggestStore  query   suggest.kt:69
14  mozilla.components.feature.fxsuggest.FxSuggestStorage$query$2   invokeSuspend   FxSuggestStorage.kt:16
15  kotlin.coroutines.jvm.internal.BaseContinuationImpl   resumeWith  ContinuationImpl.kt:9
16  kotlinx.coroutines.DispatchedTask   run   DispatchedTask.kt:112
17  kotlinx.coroutines.internal.LimitedDispatcher$Worker  run   LimitedDispatcher.kt:4
18  kotlinx.coroutines.scheduling.TaskImpl  run   Tasks.kt:3
19  kotlinx.coroutines.scheduling.CoroutineScheduler$Worker   run   CoroutineScheduler.kt:96

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox122 (nightly). We have limited time to fix this, the soft freeze is in 2 days. However, the bug still isn't assigned.

:skhan, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 10 AArch64 and ARM crashes on nightly

For more information, please visit BugBot documentation.

Comment 3

[Donal Meehan [:dmeehan]]

:bdk could you take a look at the crash, the volume is currently low but you might have some insight?

Comment 4

[Ben Dean-Kawamura [:bdk]]

Looks like it's an DB corruption issue and the question is how to handle the error without crashing firefox. I don't have much insight other than we have a similar issue with exception handling in all our components. I think the 2 options are:

• Ensure that all callers handle the exception.
• Create a wrapper layer that handles the exceptions internally and presents an exception-less API.

Not sure what's the best for Suggest, what do you think Lina?

Comment 5

[Lina Butler (ex-Mozilla)]

Oh, that's not good! We do catch SuggestApiException when we call ingest, but not when we call query.

I wrongly assumed that callers of onInputChanged would catch exceptions, based on some logging I saw in logcat—but that logging was actually coming from handlePlacesExceptions, specifically for Places.

The same code path exists in 120 and 121, so both are technically "affected". However, Firefox Suggest is only enabled by default on Nightly, which explains why we're only seeing the crash on 122 now.

Users who were enrolled in the Cross-Platform Suggest Nimbus experiment (which we stopped) would also have seen this crash when they were enrolled, though, if their database was corrupted. I wonder if that explains (at least some of) the 120 crashes in the ongoing incident.

Back to this bug: let's ensure that Suggest catches (and ignores / reports) all SuggestApiExceptions—FxSuggestStorage would be a good place to add this logic.

Comment 6

[Lina Butler (ex-Mozilla)]

Oops, mid-air collision! I was just going to say, "disabled" is more accurate than "affected", and was typing this comment out when you updated the flags.

We can get a fix into 122, before the Nightly merge, and re-target our Nimbus experiment to 122 in January. Given the holiday freeze for Nimbus experiments, and that this feature is disabled by default on Beta and Release, we aren't losing much.

Comment 7

[Donal Meehan [:dmeehan]]

Thanks :lina, the nightly soft code freeze has started but you have time to fix this before the merge day on 2023-12-18.
Otherwise we can always take an uplift!

Comment 8

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1869225 - Catch `SuggestApiException`s. (#4844)

Comment 9

[BMO Github Automation]

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/2ffae770dca04824498315203182ce7b4e44c102
[main] Bug 1869225 - Catch SuggestApiExceptions.

Comment 10

[Lina Butler (ex-Mozilla)]

Comment on attachment 9368750 [details] [review]
[mozilla-mobile/firefox-android] Bug 1869225 - Catch SuggestApiExceptions. (#4844)

Beta/Release Uplift Approval Request

• User impact if declined: Users enrolled in the Cross-Platform Suggest experiment can experience crashes when they type into the awesomebar if the database that contains search suggestions for the experiment is corrupt.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch adds code to catch and report exceptions, in a feature that's disabled by default. It's tricky to reproduce the crash manually, because it requires a specially crafted database on the device. However, the new code has good unit test coverage.
• String changes made/needed: None
• Is Android affected?: Yes

Comment 11

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1869225 - Catch `SuggestApiException`s. (backport #4844) (#4888)

Comment 12

[Lina Butler (ex-Mozilla)]

Comment on attachment 9369340 [details] [review]
[mozilla-mobile/firefox-android] Bug 1869225 - Catch SuggestApiExceptions. (backport #4844) (#4888)

Beta/Release Uplift Approval Request

• User impact if declined: Users enrolled in the Cross-Platform Suggest experiment can experience crashes when they type into the awesomebar if the database that contains search suggestions for the experiment is corrupt.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch adds code to catch and report exceptions, in a feature that's disabled by default. It's tricky to reproduce the crash manually, because it requires a specially crafted database on the device. However, the new code has good unit test coverage.
• String changes made/needed: None
• Is Android affected?: Yes

Comment 13

[Donal Meehan [:dmeehan]]

Comment on attachment 9369340 [details] [review]
[mozilla-mobile/firefox-android] Bug 1869225 - Catch `SuggestApiException`s. (backport #4844) (#4888)

Approved for Fenix 122.0b2

Comment 14

[BMO Github Automation]

Authored by https://github.com/linabutler
https://github.com/mozilla-mobile/firefox-android/commit/e473541a1f62bc25407e83c83de377f7e7b83c9d
[releases_v122] Bug 1869225 - Catch SuggestApiExceptions.

Comment 15

[Olivia Hall [:olivia]]

(Just adding Sentry link as a reference.)