Closed Bug 1483396 Opened Last year Closed 11 months ago

[Shield] Pref Flip: Search Shortcuts in Activity Stream, release 62

Categories

(Shield :: Shield Study, enhancement, P1)

enhancement

Tracking

(firefox61 unaffected, firefox62+ fixed, firefox63 unaffected)

RESOLVED FIXED
Tracking Status
firefox61 --- unaffected
firefox62 + fixed
firefox63 --- unaffected

People

(Reporter: tspurway, Assigned: tspurway)

References

(Blocks 1 open bug, )

Details

Basic description of experiment 1 (Filtering):
Remove  Alexa Top 5 search providers (Google, Yahoo, Bing, Ask, Duck Duck Go) from showing by default and organically in Top sites.  Additionally, remove the top site that corresponds to the default search engine  from the list of preloaded providers (Google, Bing, eBay, Wikipedia, Amazon, Twitter, Duck Duck Go).  

For example, if a user has Wikipedia set as their One-Click search engine in Firefox Preferences, Wikipedia and the Alexa top 5 search providers (Google, Yahoo, Bing, Ask, Duck Duck Go), will not show up in Top Sites by default and organically.

The logic for manually adding and pinning these top sites remains unchanged.



2.Basic description of experiment 2 (Search Shortcuts):
Search Shortcuts in Top Sites

Add @Amazon shortcut and @Google shortcut to all users. Once a shortcut is clicked on, the address bar gets focus and is ready to search the clicked-on source.

Note that the feature has already passed data and legal review (bug 1480513), has been code-reviewed by Firefox peers (:k88hudson :Mardak), is undergoing the usual QA oversight for Firefox features (:cmuresan), and does not require additional security review (as per :k88hudson and :Mardak).
:Matt_g - adding you for needinfo for experiment design review
Flags: needinfo?(mgrimes)
:harter - you were going to run an analysis to give us a good number for population sizing for 'exp 2'
Flags: needinfo?(rharter)
Flags: needinfo?(mgrimes)
Blocks: 1479806
A 1% sample should suffice. I've added a note to the doc as well.
Flags: needinfo?(rharter)
François, can you r+ here for the data review? I know it happened in other bugs, but it would be great to have your approval reflected here for the study.
Flags: needinfo?(francois)
Science Review: R+
[Tracking Requested - why for this release]:
Summary: [Shield] Opt-out Study: Search Shortcuts in Activity Stream → [Shield] Opt-out Study: Search Shortcuts in Activity Stream, release 62
Priority: -- → P1
Severity: normal → enhancement
Just to be clear about the percentages and targeting to be for each branch and each experiment:

Experiment 1 (Filtering): region = USA, locale = en-US, Firefox Desktop 62 Release
  - control branch = 1% (browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile=false (DEFAULT))
  - variant branch = 99% (browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile=true)

Experiment 2 (Search Shortcuts): region USA, locale = en-US, Firefox Desktop 62 Release 
  - control branch = 99% (browser.newtabpage.activity-stream.improvesearch.topSiteSearchShortcuts=false (DEFAULT))
  - variant branch = 1% (browser.newtabpage.activity-stream.improvesearch.topSiteSearchShortcuts=true)
Ryan, this is one of the studies we talked about in the dry run meeting; we'd like to launch on 9/7.
Flags: needinfo?(ryanvm)
Has QA sign-off, data, and science review. Rollout strategy looks reasonable and Fx62 is on track to go to 100% rollout tomorrow. Approving on behalf of RelMan for deployment, pending executive sign-off.
Flags: needinfo?(ryanvm)
Nick, can I get your signoff to launch this study Friday, 9/6/18? Thanks.
Flags: needinfo?(nnguyen)
approved
Flags: needinfo?(nnguyen)
Recipes are live.
Following the analysis of the data (https://docs.google.com/presentation/d/1I905E-CXuUnIdKHhi7dFtXHizg30Z92RsNbdoBuR4Rs/edit?ts=5baa9763#slide=id.g424a1c38d9_0_114) from the experiment, the team decided forward with a rollout plan. 

The details are still being worked on, expect to have a proposal by Monday, 10/1.
Depends on: 1495565
Depends on: 1495577
Summary: [Shield] Opt-out Study: Search Shortcuts in Activity Stream, release 62 → [Shield] Pref Flip: Search Shortcuts in Activity Stream, release 62
:Matt_G - in all the rush to get the Search Shortcuts Experiment 2 out, we forgot to launch Experiment 1 from this study (see comment #8).

This was to be a separate experiment (because it flipped a different pref) and be run on 1% of release and flip the pref:
browser.newtabpage.activity-stream.improvesearch.noDefaultSearchTile=false

The experiment slug should start with: pref-flip-activity-stream-search-filtering

We would really like to see the data from this, and have already done all the footwork of getting it approved and reviewed.  Would it be possible to flip this pref for the remainder of 62 release?
Flags: needinfo?(mgrimes)
That one did ship. I believe it was unintentionally flagged as high volume so it didn't show up in Test Tube. Slug: prefflip-activity-stream-search-filtering-v1-1483396
Flags: needinfo?(mgrimes)
😅 Thanks, Matt! 

:nanj, can we add some analysis to the search shortcuts dashboard to understand the effect of the filtering?
Flags: needinfo?(najiang)
(In reply to Tim Spurway [:tspurway] from comment #17)

> :nanj, can we add some analysis to the search shortcuts dashboard to
> understand the effect of the filtering?

Yep, will do. Just confirmed that we did collect metrics for this shield study.
Flags: needinfo?(najiang)
:harter - we would really like to understand the overall effect/lift on SAP usage of the 'filtering' experiment as part of our analysis.  This was the experiment where we filtered search engines out of Top Sites for 99% of the 62 population, but we held back 1% so these results weren't filtered.  The slug for this second experiment was activity-stream-search-filtering-v1-1483396

Could you run the numbers for this experiment as well?
Flags: needinfo?(rharter)
Filed Bug 1499872 to track. Sounds good. I will not be able to handle this before early November, but I'll keep you appraised of progress.
Flags: needinfo?(rharter)
Status: NEW → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.