1876123 - Add support to the Suggest Rust component for limiting suggestion types to ingest

Status: RESOLVED FIXED

(Application Services :: Suggest, task, P1)

Description

[Lina Butler (ex-Mozilla)]

In our Rust component, the SuggestionQuery.providers property lets callers specify a list of suggestion types to fetch. For example, on mobile, we currently pass at most AMP and WIKIPEDIA, and exclude Pocket, AMO, and Yelp suggestions.

However, the component still ingests all types of suggestions—AMP, Wikipedia, Pocket, AMO, and Yelp—even if the consumer never shows all of them. Limiting what types of suggestions the component ingests from Remote Settings would help us save a bit of bandwidth, disk space, and time spent for updates and first-run ingests.

Over in bug 1875909, AMP would also like us to show a different set of sponsored suggestions on mobile than on Desktop. If we could separate mobile and Desktop AMP suggestions, in Remote Settings and in our component, maybe we could avoid downloading Desktop suggestions on mobile, and vice versa? One wrinkle here is that both the Desktop AMP and Wikipedia suggestions are currently stored in the same Remote Settings record type, so I think we would need to separate those first if we wanted to get the bandwidth savings.

To make this work, we could:

• Add a SuggestIngestionConstraints.providers option, that lets consumers specify what providers (suggestion types) they'd like to download.
• Split our single last_ingest meta key, which currently holds the last_modified time of the newest Remote Settings record we ingested, into type-specific last_ingest meta keys. (Alternatively, we can turn the last_ingest field into a JSON object, keyed by the Remote Settings record type). This would let us store the last_modified times of the newest Remote Settings records we ingested of each type.
• Use those type-specific last_ingest times to filter the records that we request from Remote Settings by type.

The type-specific last_ingest timestamps would let the component request records oldest-to-newest for each type of suggestion. For example, if our mobile browsers only ingest AMP and Wikipedia suggestions now, and we want to enable Pocket, the browsers would fetch all Remote Settings records where type=pocket-suggestions&last_modified_gt=0, without having to re-ingest any AMP or Wikipedia suggestions if they didn't chage.

If the consumer indicates they only want to ingest suggestions of certain types, we can filter on last_modified and type when we make the request to Remote Settings. If they don’t specify the constraint, we can request all Remote Settings records where last_modified is newer than the oldest last_ingest timestamp—that way, we’ll catch stragglers from any other records that we haven't ingested yet.

Comment 1

[Ben Dean-Kawamura [:bdk]]

I think this is done, although I can't find the PR for it. @lina @nan should we close this one?

Comment 2

[Nan Jiang [:nanj]]

Yes, this is closed via https://mozilla-hub.atlassian.net/browse/DISCO-2640.