[Shield] Unified URL Bar - Additional testing for maxRichResults

RESOLVED FIXED

Status

P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: Paolo, Assigned: Paolo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [fxsearch])

(Assignee)

Description

2 years ago
This is about investigating a study branch with browser.urlbar.maxRichResults = 6.

Javaun and Marco can add more details as to the expected behavior in this situation.
Flags: needinfo?(mak77)
Flags: needinfo?(jmoradi)
I'd like to add another experiment branch for maxRichResults =6, biasing for speed as best we can. Worst case is that the data is bad and we scrap the cohort, but this is valuable to test early and often. 

Chromium browsers (Chrome and Opera) have 6 results (the user typed query + 5 suggestions), and this option is to start testing our experience vs. their UX as best we can currently duplicate it, to help us find our sweet spot. 


The experiment questions are:

1. What happens to search volume from the bar when we go from 10 maxRichResults (default) to 6? 
For users with no search history, they'll see their typed input plus probably 5 search suggestions. but users with heavy history (including myself) often see no search suggestions unless we type something very new or long. We might expect for users in this "6" branch to have less search volume. However, since eyetracking studies reveal that users don't look very far down the URlBar results, we may also see that even with 10 results, users aren't selecting suggestions from the URLbar. The distribution of users will be more interesting than the aggregates. (Ideally, we can separate out users with newer vs. older profiles to see how they behave)

2. What happens to usage and satisfaction? Are there patterns here for users with more vs. less history (or older vs. newer profiles).

We're going to have to test "6" in many tests, but this is a safe first look. If it's a bad UX, it's still an opt-in test with a very small number of users. 

One implementation detail: 
maxRichResults doesn't change until restart. Dave Z asked the question: can we also wait to unify the bar until restart? Otherwise users will see unification with 10 results, then after restart they'll see 6. Ideally, they see both at the same time. 

Again: let's balance all things against speed. Our idea is to get this out ASAP. Speed matters ;-)
Flags: needinfo?(jmoradi)
Whiteboard: [fxsearch]
(In reply to Javaun Moradi [:javaun] from comment #1)
> 2. What happens to usage and satisfaction? Are there patterns here for users
> with more vs. less history (or older vs. newer profiles).

This probably requires a definition for "satisfaction", how do we measure it for this case?

Btw, my questioning is broader: what do we expect to gain/lose by reducing an already small space where we already try to cramp a lot more information that can fit?
We won't gain more precise results, since the precision won't change.
We're unlikely to gain on performance, since everything is async and we plan to implement interruptible queries.

The change from 12 to 10 results had a very clear reasoning behind, we didn't want anymore a scrollbar and non-visible results, we wanted all results for which we paid a price to be visible. And we didn't want to have a huge panel covering all of content so results were reduced to a point where (with onelining) the panel was about same size as before.

If I'd know what we are trying to achieve/understand, it may be easier to outline an expected behavior.
Flags: needinfo?(mak77)
(Assignee)

Comment 3

2 years ago
(In reply to Javaun Moradi [:javaun] from comment #1)
> One implementation detail: 
> maxRichResults doesn't change until restart

This seems to take effect immediately, so we don't need to wait until restart to apply the changes.

One thing to note is that this preference may also be controlled by Sync. I have added a diagnostic measurement to check the actual value of the preference during the course of the study, so we can be confident it has not been changed.

I've also fixed some broken code paths for restoring the preference values. I've added diagnostic measurements in case the profile on which we install the experiment already has these preferences set to non-default values by a previous experiment, or manually by the user.

This version just changes the maxRichResults preference. I've made sure we are able to push updates to the experiment, so we can launch now but we can still update to something more sophisticated if needed.

I'll post the updated add-on to bug 1359118.
Assignee: nobody → paolo.mozmail
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Priority: -- → P1
You need to log in before you can comment on or make changes to this bug.