1061755 - Figure out a plan for testing user performance with awesomebar results

Status: NEW

(Firefox :: General, defect)

Description

[(Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo]

We are making lots of changes to the awesomebar right now, and there are new changes in the pipeline.
While I think (hope) that we are instrumenting all of this, but it would be good to have a plan ready to do some directed experimentation.

We need to answer these questions:
- What are our goals with the awesomebar? (e.g. Users typing as little as possible, users navigating as quickly as possible etc.)
- How do we measure how we are doing on these goals?
- What kinds of experiments, A/B tests etc. can we run to help us move towards those goals?

Comment 1

[u428464]

Bug 775825 may be relevant here.

Comment 2

[Bill Selman]

The research plan:
1. Create a special build based on prototype in Bug #1040923
2. Test qualitatively with users. Test protocol is already created.
3. Add telemetry probes to measure autocomplete usage.

Comment 3

[u428464]

This bug was originally created to test bug 583683. So it was more about question like knowing if the users click on the results that are hidden by the scrollbar or how many entries it is the most efficient to show (with or without a scrollbar).

Comment 4

[Bill Selman]

(In reply to Guillaume C. [:ge3k0s] from comment #3)
> This bug was originally created to test bug 583683. So it was more about
> question like knowing if the users click on the results that are hidden by
> the scrollbar or how many entries it is the most efficient to show (with or
> without a scrollbar).

We read this bug as more general about changes to the Awesome Bar, not just the optimal average number of non-scroll entries. In discussion, we decided to expand the scope to focus on a larger set of changes. 

However, we can include a study of this specific research question. Or we can write a separate bug to focus on the specific question of entries. That said, I know that Gregg Lind has been working on this question.

Comment 5

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

This is a huge topic :)  Glad to be involved.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.