1893086 - Port the potential impressions framework to the suggest component

Status: RESOLVED FIXED

(Application Services :: Suggest, task, P1)

Description

[Drew Willcoxon :adw]

Background and motivation

On desktop we recently implemented a potential-impressions experiment framework that lets us run Nimbus experiments to gather usage data on keywords without making any code changes (bug 1881873, spec).

Management has raised the idea of using the framework for Fakespot to measure potential Fakespot suggestion impressions before we spend a lot of time and resources implementing them, but there are a few technical problems.

The major problem is the framework expects keywords to be defined in a Nimbus variable, but there is a very large number of Fakespot keywords. The framework wasn't designed with such a large number in mind. Could we put tens or hundreds of thousands of keywords in the variable? According to Barret, there's no hard limit on the size of this particular variable, so maybe? But it definitely doesn't seem appropriate. (There's a hard limit for features that set isEarlyStartup and variables that use setPref, neither of which applies here.)

A minor problem is the framework does our usual simplistic exact keyword matching, but we may end up using fts5 for Fakespot. This isn't a huge problem on its own because we could always manually tokenize the suggestions data to generate a final list of keywords, and then give that final list to the framework and let it do exact matching.

Another minor problem is the framework is desktop only. I don't think we intend to do Fakespot on mobile right now, but we might in the future. Regardless of Fakespot, it would be nice if mobile could use as much of the framework as possible.

Implementation

I'd like to re-implement the framework in the Rust component using remote settings for keywords. That will let us store arbitrarily large keyword sets, use the same matching strategy(ies) we use for real suggestions, and allow mobile to use it. We can't implement the entire framework in the component because telemetry and Nimbus will be platform specific (I think?), but we can implement the keywords and core matching parts.

My implementation strategy is:

• Introduce a "phantom" suggestion type. (I'm not married to the name.) [Edit: I ended up calling them "exposure" suggestions in the final implementation.] The idea is that these suggestions pretty much behave like real suggestions and are matched the same, but they aren't intended to be shown to the user. We can introduce new types of phantom suggestions without making any code changes.
• In remote settings:
  • All phantom suggestion records have the type phantom-suggestions.
  • A record's attachment can hold one or more phantom suggestions.
  • Each suggestion has a type string and keyword data. The type is unique across phantom suggestions. The keyword data is TBD and will depend on how we do matching. It might be a list of keywords for exact matching (like AMP), it might be a list of keywords where we do prefix matching on the client (like Pocket and addons), it might be a list of titles that we tokenize on the client (if we use fts5 for example).
  • If a suggestion's keyword data is very large, the suggestion can be split up into multiple records to keep attachment sizes small. In that case, the suggestion in each attachment should use the same type to indicate the data represents the same logical suggestion.
• On the client:
  • The suggest component has a single Phantom provider and treats all phantom suggestions the same regardless of individual suggestion type. i.e., there's at most one details table for all phantom suggestions, not per-type tables, and ingest and matching is the same. (Although in the future I can imagine consumers being able to specify a matching strategy if that's helpful, e.g. exact, prefix, fts.)
  • Introduce an optional SuggestionQuery string member called phantom_suggestion_type. A query will return all phantom suggestions that match that type and the query's keyword along with all the usual suggestions that match the keyword.
  • The Phantom suggestion type returned to consumers includes a matched_keyword string whose value is the full keyword that matched the query. Hopefully identifying the full keyword remains possible with more sophisticated matching strategies we might implement.

Consumers

The required one-time code changes in suggest consumers are:

• Add a Nimbus variable whose value will be the intended phantom suggestion type in each experiment.
• Add telemetry event(s) or other metrics for potential exposures/impressions.
• When the Nimbus variable is defined, pass it to queries using phantom_suggestion_type. When the suggest component returns a phantom suggestion, record the telemetry.

To test a new set of keywords, we would do this:

• Come up with a type name to describe the set. e.g., "yelp", "fakespot", "sports", "finance"
• Upload a new phantom suggestion to RS with the keywords and type. If the keyword set is very large, it can be split over multiple records as mentioned above.
• Create a new experiment and set the Nimbus variable to the type.

Comment 1

[BMO Github Automation]

Attachment: [mozilla/application-services] Bug 1893086 - Port the potential impressions framework to the suggest component (#6343)

Comment 2

[Drew Willcoxon :adw]

The PR was merged so I'll close this but it hasn't been vendored into m-c yet.

Comment 3

[Drew Willcoxon :adw]

This was vendored in bug 1916530.