1518829 - Define north star metric[s] for AwesomeBar tuning

Status: RESOLVED FIXED

(Data Science :: General, task, P1)

Description

[Ryan Harter [:harter]]

The search team is planning to make some big improvements to the AwesomeBar in 2019. We need a quality metric to verify our changes are improving the user experience.

For context, here are some characteristic changes we may pursue in 2019:

• Re-rank results in the AwesomeBar, possibly using florian's federated learning proof of concept
• Changing the order or priority of different result type blocks (history, searches, bookmarks)
• Changing auto-complete behavior

We want to have one north-star metric that will measure "quality" with high confidence. I.e. if the metric shows an improvement we want to be confident it's actually an improvement to the user experience. It's OK if we sacrifice some sensitivity to this end.

If we do need to sacrifice sensitivity, we should have some supplementary metrics with higher sensitivity. We can then calibrate the supplementary metrics against the north-star metric to allow us to measure smaller changes.

The goal for Q1 is to have a north-star metric defined with proposals for any necessary supplementary metrics. We'll also need to specify any additional telemetry we'd like to collect.

We can (and will probably need to) improve these metrics once we start experimenting with new features. We want to have a head start on these upcoming improvements though.

Some rough metric-outlines we discussed in Orlando:

• Awesomebar engagements - e.g. number of time completes a query through the AB
• User issued keystrokes per engagement
• Searches volume from AB
• Mean accepted result position - e.g. the position of the result a user clicks
• For AwesomeBar ranking changes we can explore reciprocal rank, but this is likely future work.

Comment 1

[Teon Brooks [:teon]]

Attached is the North Star Metric proposal for the AwesomeBar. This has been shared with key stakeholders for comments and feedback.

https://docs.google.com/document/d/1ZcOuQVg_jTd8YLtMyaxiW5ikuxvs3zAqrrEqkGC4S0Q/edit?usp=sharing