Closed Bug 1674187 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: Holdback Study for Quantumbar v2 Fx 82.0 to 85.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox82+ fixed, firefox83+ fixed, firefox84+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox82 + fixed
firefox83 + fixed
firefox84 + fixed

People

(Reporter: experimenter, Assigned: nraketic)

References

(Depends on 1 open bug,
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

browser.urlbar.experiment.update2

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Holdback Group, No Update 50%:

Value: false

Holdback group that does not get v2 update
        
- Control QuantumBar v2 Group 50%:

Value: true

Group that gets v2 update
        

    What version and channel do you intend to ship to?

10% of Release Firefox 82.0 to 85.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Nov 10, 2020 - Feb 22, 2021 (104 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will measure `search_counts`, `n_search_alias`, `n_searchmode`, and `search_engagement`, which is `sap/(organic + sap)`.
We want to know the search volume impact of the new features for the address bar. Search engagement and QuantumBar searchmode engagement (proxy for revenue) are the KPIs.

# Metrics
- `search_counts`
- `searchmode_counts`
- `search_counts_by_sap`
- `search_engagement`  = sap / (organic + sap)
- Amazon search mode count by entry (oneoffs, typed) for treatment branch
- Amazon search alias count for control branch

Special Note:

- Analysis should be mindful of the TopSites study that will be running at the same time as this study.

- From Engineering: `urlbar.tips.tabtosearch_onboard-shown` , `urlbar.picked.searchmode.keywordoffer`, `urlbar.picked.searchmode.topsites_urlbar`, `urlbar.picked.searchmode.tabtosearch`, and `urlbar.picked.searchmode.tabtosearch_onboard` are broken in Fx 83. All the other urlbar.picked.searchmode.* probes are fine

    Who is the owner of the data analysis for this experiment?

tbrooks@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:



    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/holdback-study-for-quantumbar-v2/

Holdback Study for Quantumbar v2

Holdback Study - Release QuantumBar v2 to 90% of users and 5% treatment, 5% control for a holdback study

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-364
More information: https://experimenter.services.mozilla.com/experiments/holdback-study-for-quantumbar-v2/

Blocks: urlbar-searchshortcuts
No longer blocks: 1647921

Holdback Study - Release QuantumBar v2

Firefox 82 to Firefox 85

We have finished testing the [PI-843] Holdback Study - Release QuantumBar v2 experiment on Firefox 82 and 83. Considering the fact that the experiment will be live until Firefox 85, for a more accurate assessment, we will retest the experiment on Firefox 84 and 85 builds closest to RC / Firefox Release RC build and send a new sign off email for each version.

Quality status: GREEN - SHIP IT

Why is this experiment green?

Delivery mechanism:

  • During testing we have encountered one new issue:
  • Bug 1675796 - The profile remains enrolled in the "QuantumBar v2 Group" branch after manually disabling the feature from about:config
  • This is an edgecase because users would have to manually disable the feature using the "about:config" page. However, QA is concerned that this might also affect the experiment’s telemetry data since the feature will disabled but they will still be enrolled in the experiment.

Product specific:

  • Since the features functionality was not in scope, we only verified that the features are enabled/disabled accordingly by the experiment and have found no issues during testing.
    • The QuantumBar v2 features were tested and signed off as GREEN by the Desktop teams (PI-715 and PI-735).

Testing Summary:

  • Test Runs: Test Rail;

Tested Platforms:

  • Windows 10 x64,
  • macOS 10.15.6;
  • Ubuntu 20.04 x64;

Tested Firefox versions:

  • Firefox Release 82.0.2 en-US (Build ID: 20201027185343)
  • Firefox Release 82.0.2 de (Build ID: 20201027185343)
  • Firefox Release 82.0.2 fr (Build ID: 20201027185343)
  • Firefox Beta 83.0b9 en-US (Build ID: 20201105203649)
  • Firefox Beta 83.0b9 de (Build ID: 20201105203649)
  • Firefox Beta 83.0b9 fr (Build ID: 20201105203649)

[Tracking Requested - why for this release]:

tracking-firefox82: --- → ?
tracking-firefox83: --- → ?
tracking-firefox84: --- → ?
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Holdback Study for Quantumbar v2 → [Experiment] Pref-Flip Experiment: Holdback Study for Quantumbar v2 Fx 82.0 to 85.0 Release
Start Date: 2020-11-10 End Date: 2021-02-22
Severity: -- → S4
User Story: (updated)
Whiteboard: [no-nag]

Holdback Study - Release QuantumBar v2
Firefox 82 to Firefox 85

We have finished testing the “[PI-887] - Holdback Study - Release QuantumBar v2” experiment on Firefox 84.

The study is already live, but since it's running until Firefox 85, we are running our test suite on the latest builds of each Beta cycle to make sure that there are no new issues between Firefox versions.

Quality status: GREEN

Why is this experiment green?

  • We haven’t found any issues during testing.

Testing Summary:

Tested Platforms:

  • Windows 10 x64;
  • macOS 10.16.6;
  • Ubuntu 20.0 x64;

Tested Firefox versions:

  • Firefox Release 82.0.3 en-US (Build ID: 20201108180448)
  • Firefox Release 82.0.3 de (Build ID: 20201108180448)
  • Firefox Release 82.0.3 fr (Build ID: 20201108180448)
  • Firefox Beta 84.0b7 en-US (Build ID: 20201201213706)
  • Firefox Beta 84 RC de build (Build ID: 20201207203640)
  • Firefox Beta 84 RC fr build (Build ID: 20201207203640)

Holdback Study - Release QuantumBar v2
Firefox 82 to Firefox 85

We have finished testing the “[PI-909] - Holdback Study - Release QuantumBar v2 for Firefox 85” experiment.
The study is already live, but since it's running until Firefox 85, we are running our test suite on the latest builds of each Beta cycle to make sure that there are no new issues between Firefox versions.

Quality status: GREEN

Why is this experiment green?

  • We haven’t found any issues during testing.

Testing Summary:

Tested Platforms:

  • Windows 10 x64;
  • macOS 10.16.6;
  • Linux Mint 20 6x4;

Tested Firefox versions:

  • Firefox Beta 85.0b8 fr (Build ID: 20210112185508)
  • Firefox Beta 85.0b9 de build (Build ID: 20210114193053)
  • Firefox Beta 85 RC en-US build (Build ID: 20210118153634)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.