1863661 - Reingest suggestions when the Firefox Suggest database is recreated

Status: RESOLVED FIXED

(Fenix :: Search, task)

Description

[Lina Butler (ex-Mozilla)]

We've had to make a handful of backward-incompatible schema changes to the Suggest database while working on the Firefox Suggest integration. Since the database doesn't contain any user data, we've handled these migrations by marking the old database as corrupt. This throws the old database away, and creates a fresh, empty one with the latest schema.

When we create (or recreate) a fresh database, we don't trigger an ingestion from Remote Settings right away. Instead, we rely on the platform's background work scheduling mechanism to ingest new suggestions "later". On Android, we use the WorkManager library to accomplish this. At application startup, FxSuggestIngestionScheduler schedules our FxSuggestIngestionWorker to run every 24 hours, when the device is on an unmetered network connection, has space, and is charged.

If the worker is being scheduled for the first time—like when a user is enrolled in the Enhanced Cross-Platform Suggest experiment, and launches Firefox 120+ for the first time—and the other constraints are met, the worker will run "soon" after startup, and then every 24 hours after that.

The problem is when we make a backward-incompatible schema change for an existing user. In that case, we'll throw away the database when it's first opened, but we won't ingest any suggestions until the next time the worker runs. So, if the worker ran 12 hours ago, it'll be at least another 12 hours before we download any suggestions from Remote Settings. And if the worker just ran an hour ago, it'll be at least 23 hours. Between the time we threw away and recreated the database, and the next time the ingestion worker runs, the user won't see any sponsored or Wikipedia suggestions at all.

Comment 1

[Lina Butler (ex-Mozilla)]

A couple of ideas for possible ways to fix this:

1. Use ExistingPeriodicWorkPolicy.REPLACE instead of ExistingPeriodicWorkPolicy.KEEP if the last ingestion time isn't set when scheduling the worker. This will force ingestion to run "soon" on first run (where there's no existing work scheduled), and when the database is recreated (that is, when there is existing work scheduled, but there's no last ingest time).
2. Unconditionally use ExistingPeriodicWorkPolicy.REPLACE when scheduling the worker. This will force ingestion to run "soon" after every app launch, and then every 24 hours after that, until the next app relaunch.

(1) is more complicated: we'd need to add a method to SuggestStore, expose it to Kotlin via UniFFI, and ensure we're not calling it from the main thread when we schedule the worker, which would probably mean turning FxSuggestIngestionScheduler.startPeriodicIngestion into a suspend function that calls our new method to decide which ExistingPeriodicWorkPolicy to use.

But (1) would keep the number of Remote Settings requests low: once every 24 hours, or after a Firefox update that makes a backward-incompatible schema change.

(2) is a lot simpler, but would mean making a request to Remote Settings on every Firefox launch. That might be OK, though!

Comment 2

[Chris Peterson [:cpeterson]]

Setting "enhancement" and "task" bug types to Severity N/A.

Comment 3

[Ben Dean-Kawamura [:bdk]]

Could we do both?

• Keep the current periodic task the same, except use setInitialDelay so that it waits a day before running the first time.
• Add a new task that calls enqueueUniqueWork() to queue up a one-time task that runs ingestion immediately, but only if the database is empty.

Comment 4

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1863661 - Reingest suggestions on startup. r=lina

Added one-time work that runs ingestion, but only if there are no
suggestions in the database. Made added an initial delay to the current
periodic work. The result is that:

• We always run ingestion on startup if the suggestions table is
  empty. This covers first-time startup and also startup after the
  suggestions table was cleared because we ran an update that changed
  the schema.
• We also run ingestion every 24 hours. This covers keeping the
  database up-to-date after the initial ingestion.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c19461920151
Reingest suggestions on startup.  r=lina,android-reviewers,Roger

Comment 6

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/c19461920151