Closed Bug 1577291 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip: activity-stream-pkt-new-tab-release69-layout-holdback-1573312 Fx 69.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox69+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox69 + fixed

People

(Reporter: experimenter, Assigned: digitalnewslab)

References

(
URL
)

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.enabled

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment discovery-stream-default 50%:

Value: true

Discovery Stream default experience (7 rows and 3 columns)
        
- Treatment activity-stream-default 50%:

Value: false

Activity Stream default experience (1 row and 3 columns)
        

    What version and channel do you intend to ship to?

6% of Release Firefox 69.0

    Are there specific criteria for participants?

Prefs:

Experiments:
All experiments under namespace 729 -  https://normandy-namespaces.web.app/?namespace=729

1. 18 Branch - https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66-confirmation-test/
2. 4 Branch -  https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66layoutstest1/
3. 2 Branch - https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66smoketest2/
4. Syndicated Boost - https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtab-1564334-release67syndicatedboost
5. https://experimenter.services.mozilla.com/experiments/smoketest-discoverystream-pref-V1
6. https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabH12019-68-2-branch-confirmation-test-V2 [Bug 1566212]
7.  Pref-Exp: AS Pocket New Tabs - Syndicated Boost [Bug 1555451]
8. Activity Stream Pocket New Tab 2-Branch Confirmation [Bug 1557131]


Any additional filters:
Countries: United States of America (US) 

Locales: English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Sep 03, 2019 - Oct 22, 2019 (49 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We are looking for less than 1% or no regression on Search volume.

    Who is the owner of the data analysis for this experiment?

Ben Miroglio [:bmiroglio]

    Will this experiment require uplift?

False

    QA Status of your code:

GREEN

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-new-tab-release69-layout-holdback-1573312/

activity-stream-pkt-new-tab-release69-layout-holdback-1573312

With the 4 Branch Confirmation Study described on bug 1548366 we validated significant gains in engagement with Pocket stories and SPOCS by enabling new layouts in the New Tab. We plan to ship this new experience in 69 on release and we would like to understand the impact this layout will have on Search volume.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-new-tab-release69-layout-holdback-1573312/

[Tracking Requested - why for this release]:

Holdback study to launch with Fx 69 for supporting the Pocket New Tab - en-US/US geo launch. \o/

tracking-firefox69: --- → ?

Holdback Study - 3 Sept - Fx 69 - Pocket New Tab
Firefox 69 Release

We have finished testing the Holdback Study - 3 Sept - Fx 69 - Pocket New Tab experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We have not found any new issues during testing the experiment, other than the known one encountered while the feature was previously tested (Bug 1524294).

Testing Summary:

  • Link to Test Suite: Link.

Tested Platforms:

  • Windows 10
  • macOS 10.14
  • Arch Linux 5.10

Tested Firefox versions:

  • Firefox RC 69
User Story: (updated)
Summary: [Experiment]: Pref-Flip: activity-stream-pkt-new-tab-release69-layout-holdback-1573312 → [Experiment] Pref-Flip: activity-stream-pkt-new-tab-release69-layout-holdback-1573312 Fx 69.0 Release
Start Date: 2019-09-03 End Date: 2019-10-22
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox69: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.