Closed
Bug 1495565
Opened 6 years ago
Closed 6 years ago
[Shield] Pref Flip Study: Search Shortcuts in Activity Stream, release 62 - Part 2 (New Locales)
Categories
(Shield :: Shield Study, defect)
Shield
Shield Study
Tracking
(firefox62+ fixed)
RESOLVED
FIXED
People
(Reporter: tspurway, Assigned: tspurway)
References
Details
(Whiteboard: [shield-enabled])
We would like to extend the first Shield study (#1483396) with exactly the same expectations, release and parameters with the following Shortcuts to the following regions: Amazon + Google shortcuts: Germany Japan United Kingdom France Italy Austria Yandex only: Russia Google only: Poland Brazil India Indonesia Spain Canada The expected outcome is that we see comparable results that we saw in the parent study. For Amazon, we saw a 300% increase in overall searches in the US. For this study we would expect at least a 100% increase in SAP searches. For Google and Yandex, we should stick with our original expectations, which was a 0.5% increase in SAP searches over control. :rharter, what would our sample size need to be to fit these expectations across all of these regions?
Assignee | ||
Updated•6 years ago
|
Flags: needinfo?(rharter)
Flags: needinfo?(mgrimes)
Comment 1•6 years ago
|
||
5% samples should allow us to detect a 1% change in overall search. Let's start there. If we aim to identify a change of 0.5% we'd need 4x the enrollment. Matt, can we expand the experiment sample if the experiment proves to be underpowered?
Flags: needinfo?(rharter)
Assignee | ||
Comment 2•6 years ago
|
||
:mcoman, I am assigning you to provide QA sign-off for this (let me know if you're available) :osunick, I am assigning you for VP sign-off
Flags: needinfo?(nnguyen)
Flags: needinfo?(marius.coman)
Comment 3•6 years ago
|
||
[Tracking Requested - why for this release]:
status-firefox62:
--- → affected
tracking-firefox62:
--- → ?
Updated•6 years ago
|
Comment 4•6 years ago
|
||
No concerns from relman for a 1% rollout for selected locales/countries (assuming QA and VP sign-off).
Flags: shield-relman+
Assignee | ||
Comment 5•6 years ago
|
||
Note that the list of Regions has been updated (Austria has been removed): Amazon + Google shortcuts: Germany Japan United Kingdom France Italy Yandex only: Russia Google only: Poland Brazil India Indonesia Spain Canada
Assignee | ||
Comment 6•6 years ago
|
||
Just to be clear, we would like to target Oct 8, 2018 for the start date of this study in Release 62.
Comment 7•6 years ago
|
||
Search Shortcuts & Improvements on New Tab - Part 2 (New Locales) Targeted: Firefox Release 62 We have finished testing the Search Shortcuts & Improvements on New Tab (New Locales). QA’s recommendation: GREEN - SHIP IT Reasoning: - We haven't found any issues during testing. Testing Summary: - Full Functional test suite: https://goo.gl/zbapia Tested Platforms: - Windows 10 x64 - Mac 10.13 - Arch Linux 4.16.6 x64 Tested Firefox versions: - Firefox 62.0.3 - Firefox 62.0.2 - Firefox 62.0.1
Flags: needinfo?(marius.coman)
Comment 9•6 years ago
|
||
we need 2 weeks of enrollment. Pause the study on Monday, October 22 (so no New people are enrolled). We want people who are enrolled to stay in the experiment even when they upgrade to Firefox 63 (which will start rolling out on October 23).
Comment 10•6 years ago
|
||
Make a decision after 3 weeks into the study. Go/No Go decision the week of October 29th We want to leave everyone in the study until November 12th.
Assignee | ||
Comment 11•6 years ago
|
||
For the Shield Slug/ID, let's make sure we name it with a prefix that is: pref-flip-activity-stream- This will ensure that both UT and Tiles pipelines will get data routed correctly
Updated•6 years ago
|
Flags: needinfo?(mgrimes)
Summary: [Shield] Opt-out Study: Search Shortcuts in Activity Stream, release 62 - Part 2 (New Locales) → [Shield] Pref Flip Study: Search Shortcuts in Activity Stream, release 62 - Part 2 (New Locales)
Engineering question: - I am unclear about the interaction between the main pref and `browser.search.region`. This is affecting the launch recipe. Can an implementor help walk through * who should be targeted and * how to trigger the effect (sounds like a combo of two prefs?) * what percentage for each region. I hope that it's a simple as: - same percentage in each country - just change the one pref (browser.search.region is autoset). Thanks!
Flags: needinfo?(tspurway)
Flags: needinfo?(mpopova)
Enabled. To disable, disable #598.
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(tspurway)
Flags: needinfo?(mpopova)
Resolution: --- → FIXED
Whiteboard: [shield-enabled]
Comment 14•6 years ago
|
||
This is still enabled but adding new users is now paused.
Comment 15•6 years ago
|
||
We've ended this study with Maria's approval.
Updated•5 years ago
|
You need to log in
before you can comment on or make changes to this bug.
Description
•