Closed Bug 1603606 Opened 6 years ago Closed 6 years ago

[Experiment] Pref-Flip Experiment: Separate Search default in Private Browsing Fx 71.0 to 73.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox71 ?)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox71 --- ?

People

(Reporter: experimenter, Assigned: mconnor)

References

(Depends on 1 open bug, Blocks 1 open bug,
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.search.separatePrivateDefault.ui.enabled,browser.search.separatePrivateDefault.ui.banner.max

    What are the branches of the experiment and what values should
    each branch be set to?

- Control Control 34%:

Value: None

Standard Search Branch
        
- Treatment Treatment NO Banner 33%:

Value: None

Will Have NOT have a banner that tells user of this feature in PBM, however the option is available
        
- Treatment Treatment Banner 33%:

Value: None

Will Have a banner that tells user of this feature in PBM
        

    What version and channel do you intend to ship to?

0.2% of Release Firefox 71.0 to 73.0

    Are there specific criteria for participants?

No filtering
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Dec 17, 2019 - Feb 15, 2020 (60 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

```
What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success?
```
We have three different branches:
- control
- ui enabled with no banner
- ui enabled with banner

We want to see what proportion of the users self-discover the option for separate private browsing mode, and what proportion of users select a separate private browsing mode after being prompted.

We will be looking at the percentage change in the search volume for sap searches and organic searches. We are interested to see if there is a recapture of in our sap from otherwise organic searches for searches conducted in the separate private browser setting.
We also would like to monitor to see if there is an effect on our sap search volume as a result of the separate private browsing mode setting.

The metrics will be using are 
- sap search counts
- organic search counts

    Who is the owner of the data analysis for this experiment?

tbrooks@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

https://jira.mozilla.com/browse/PI-418
 FYI: feature was tested in Fx71 ( preffed off),  not sure if any of that testcases can be utilized here.  Only mentioned in case it helps.

This experiment was signed off as YELLOW. For further information see: https://experimenter.services.mozilla.com/experiments/separate-search-default-in-private-browsing/#comment563

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/separate-search-default-in-private-browsing/

Separate Search default in Private Browsing

We want to test this feature (having a separate default in Private Browsing) to gain an understanding, if there is any, revenue impact.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/separate-search-default-in-private-browsing/

Experiment: Sep search eng. In Private Browsing mode

Targeted: Firefox Release 71-73

We have finished testing the Sep search eng. In Private Browsing mode experiment.

QA’s recommendation: YELLOW - SHIP IT CONDITIONALLY

Reasoning:

  • Due to the short amount of testing time we were constrained in testing only 1 version using 1 locale on 1 OS (Firefox Release 71 en-US locale on Windows 10 x64, Firefox Beta 72.0b7 fr locale on macOS 10.14.1, and Firefox Nightly 73.0a1 de locale on Ubuntu 18.04).

  • Considering that the feature on which the experiment is based was fully tested only on Firefox Release 71 and that changes might be added to Firefox Beta and Firefox Nightly which might affect the results of the tests, we cannot guarantee that no issue will appear on Fx72 and Fx73 Firefox versions when they are released.

Testing Summary:

  • Test Suites - TestRail

  • We have also created a Normandy recipe on the Stage environment and verified that clients are enrolled correctly in each branch.

Tested Platforms:

  • Windows 10 x64

  • Mac OS 10.14.6

  • Ubuntu 18.04

Tested Firefox versions:

  • Firefox Release 71

  • Firefox Beta 72.0b7

  • Firefox Nightly 73.0a1

Regards,

Robert

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Separate Search default in Private Browsing → [Experiment] Pref-Flip Experiment: Separate Search default in Private Browsing Fx 71.0 to 73.0 Release
Start Date: 2019-12-17 End Date: 2020-02-15
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.