1390592 - [Shield] Unified Search Bar Study: Round 3 Opt-in Tracking

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Shilpi Gupta [:shilpi]]

+++ This bug was initially created as a clone of Bug #1387245 +++

1. Testing result composition: We don't yet know the optimal result composition in a unified Awesome Bar. We're inclined to believe it's not what we have now (10 results, history first). We need ASAP to test that in Shield.  Our ENG plan all along was to land pref'able options prior to 57 and that we could use off-the-trains tech (i.e. Shield) to experiment with composition and even work right up to November to deploy the final best option via Shield, if that's needed.

Panos team landed new pref-able options in 56 to allow us to put search suggestions on top (right now they're only bottom) and also to allow us to vary the number of Awesome Bar results (such as 5 total results) where a user doesn't just see 5 history results, but sees at least a few search too. 

    
Branches for Testing Result Composition and Unified vs. 2 Bar preferences:

    Control: Normal Fx (two separate bars, as they exist now)
    Unified Bar with 10 results, history on top (currently default under unification)
    Unified Bar with 10 results, search on top
    Unified Bar with 5-6 results, history on top
    Unified Bar with 5-6 results, search on top
    Unified Bar with 10 results, history on top (currently default under unification) + top hit on top
    Unified Bar with 10 results, search on top + top hit on top
    Unified Bar with 5-6 results, history on top + top hit on top
    Unified Bar with 5-6 results, search on top + top hit on top


2. Testing to understand who gets unified vs. keeps search bar.

Comment 1

[Shilpi Gupta [:shilpi]]

Paolo is working on the add-on. Expected to be ready shortly. QA working on testing. 
Consent page will be reused from v2 unified search shield study. Legal bug will be reused too since no new information is being collected. 

We will launch this study as opt-in.

Comment 2

[:Paolo Amadini]

Shilpi, we're doing QA and branch definition for the opt-in study in bug 1387245. Since all the information is already there, can we keep using that bug number and change the title based on the new plan?

Comment 3

[Shilpi Gupta [:shilpi]]

Sure Paolo, I would not want to disrupt the ongoing conversations. I will keep this bug alive for tracking status of opt in study. Noted that the background work on the addon + QA will be largely reused for both optin and optout experiments.

Let me link the 2 related bugs. Thanks for flagging!

Best
Shilpi

Comment 4

[Matt Grimes [:Matt_G]]

Hey folks. Can we get a quick recap of the results and then have this bug closed? A couple sentences is fine.

Comment 5

[Dave Zeber [:dzeber]]

AKA the v3.2 Unification study.

Here's a summary of the findings:

- No significant differences in terms of engagement or retention (in particular page views)
- All unification branches saw a drop in search rates of about 3-8% relative to control
- Unification branches with search suggestions on top performed the best
- No clear differences between 10 and 7 results in the dropdown

I can provide more details or a link to the report on request.