Closed Bug 1495565 Opened 6 years ago Closed 6 years ago

[Shield] Pref Flip Study: Search Shortcuts in Activity Stream, release 62 - Part 2 (New Locales)

Categories

(Shield :: Shield Study, defect)

defect
Not set
normal

Tracking

(firefox62+ fixed)

RESOLVED FIXED
Tracking Status
firefox62 + fixed

People

(Reporter: tspurway, Assigned: tspurway)

References

Details

(Whiteboard: [shield-enabled])

We would like to extend the first Shield study (#1483396) with exactly the same expectations, release and parameters with the following Shortcuts to the following regions:

Amazon + Google shortcuts:
Germany
Japan
United Kingdom
France
Italy
Austria

Yandex only:
Russia 

Google only: 
Poland
Brazil
India
Indonesia
Spain
Canada

The expected outcome is that we see comparable results that we saw in the parent study.  

For Amazon, we saw a 300% increase in overall searches in the US. For this study we would expect at least a 100% increase in SAP searches.  For Google and Yandex, we should stick with our original expectations, which was a 0.5% increase in SAP searches over control.

:rharter, what would our sample size need to be to fit these expectations across all of these regions?
Flags: needinfo?(rharter)
Flags: needinfo?(mgrimes)
5% samples should allow us to detect a 1% change in overall search. Let's start there. If we aim to identify a change of 0.5% we'd need 4x the enrollment. Matt, can we expand the experiment sample if the experiment proves to be underpowered?
Flags: needinfo?(rharter)
:mcoman, I am assigning you to provide QA sign-off for this (let me know if you're available)

:osunick, I am assigning you for VP sign-off
Flags: needinfo?(nnguyen)
Flags: needinfo?(marius.coman)
[Tracking Requested - why for this release]:
No concerns from relman for a 1% rollout for selected locales/countries (assuming QA and VP sign-off).
Flags: shield-relman+
Note that the list of Regions has been updated (Austria has been removed):

Amazon + Google shortcuts:
Germany
Japan
United Kingdom
France
Italy

Yandex only:
Russia 

Google only: 
Poland
Brazil
India
Indonesia
Spain
Canada
Just to be clear, we would like to target Oct 8, 2018 for the start date of this study in Release 62.
Search Shortcuts & Improvements on New Tab - Part 2 (New Locales)

Targeted: Firefox Release 62


We have finished testing the Search Shortcuts & Improvements on New Tab (New Locales).


QA’s recommendation:  GREEN - SHIP IT


Reasoning:

- We haven't found any issues during testing.


Testing Summary:

- Full Functional test suite: https://goo.gl/zbapia


Tested Platforms:

- Windows 10 x64

- Mac 10.13

- Arch Linux 4.16.6 x64


Tested Firefox versions:

- Firefox 62.0.3

- Firefox 62.0.2

- Firefox 62.0.1
Flags: needinfo?(marius.coman)
Approved.
Flags: needinfo?(nnguyen)
we need 2 weeks of enrollment.
Pause the study on Monday, October 22 (so no New people are enrolled).
We want people who are enrolled to stay in the experiment even when they upgrade to Firefox 63 (which will start rolling out on October 23).
Make a decision after 3 weeks into the study.
Go/No Go decision the week of October 29th
We want to leave everyone in the study until November 12th.
For the Shield Slug/ID, let's make sure we name it with a prefix that is:

pref-flip-activity-stream-

This will ensure that both UT and Tiles pipelines will get data routed correctly
Flags: needinfo?(mgrimes)
Summary: [Shield] Opt-out Study: Search Shortcuts in Activity Stream, release 62 - Part 2 (New Locales) → [Shield] Pref Flip Study: Search Shortcuts in Activity Stream, release 62 - Part 2 (New Locales)
Engineering question:  
- I am unclear about the interaction between the main pref and `browser.search.region`.  This is affecting the launch recipe.  Can an implementor help walk through 

* who should be targeted and
* how to trigger the effect (sounds like a combo of two prefs?)
* what percentage for each region.

I hope that it's a simple as:
- same percentage in each country
- just change the one pref (browser.search.region is autoset).

Thanks!
Flags: needinfo?(tspurway)
Flags: needinfo?(mpopova)
Enabled.

To disable, disable #598.
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(tspurway)
Flags: needinfo?(mpopova)
Resolution: --- → FIXED
Whiteboard: [shield-enabled]
Depends on: 1500556
This is still enabled but adding new users is now paused.
We've ended this study with Maria's approval.
You need to log in before you can comment on or make changes to this bug.