1573312 - [Pocket] Newtab Experiments - Hold Back Study in Desktop Firefox 69 for New Layouts

Status: RESOLVED FIXED

(Data Science :: General, task)

Description

[Tawanda Kanhema]

Brief Description of the request (required):

Hold Back Study to verify long term impact of New Tab layouts on Firefox browser metrics - Search Volume, Total Hours, URIs

Business purpose for this request (required):

Measure the effect of the new layouts on search volume to determine impact on search revenue.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Run in 69 cycle starting on September 10. This Hold Back study will help inform decisions on whether the new layouts improve the user experience over a longer period of time.

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

Experiment Design Document https://docs.google.com/document/d/16IGZHRFSCTppSrt7NagDw9-LB2iORQFQc8vidlndBZo/edit?usp=sharing

Name of Data Scientist (If Applicable):

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Jessilyn Davis]

Experiment Design Document - New Tab Hold Back Study in Fx 69:
https://docs.google.com/document/d/16IGZHRFSCTppSrt7NagDw9-LB2iORQFQc8vidlndBZo/edit?ts=5d55d5cd

Drafted Experimenter Setup:
https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-new-tab-release69-layout-holdback-1573312/

Technical documentation for the feature:
https://docs.google.com/document/d/1MvSTug5nb9WY7kf-GY2gkbhXIpTFovkVGjNPbN_Kjyw/edit#

This study meets the criteria for the Holdback Study 1: The feature is controlled by a preference and being turned on via Normandy Rollout.

• This is the ideal case. Since the study and the rollout happen with the same system, they can be coordinated to make sure that users either get the rollout, or the study, and not both.
  (https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=FIREFOX&title=Study+FAQ#StudyFAQ-Holdbackstudies)

Timing:
Ideally, this study will start and finish enrollment before Fx 69 releases on Tuesday, Sept 3. That means we'd like to target enrollment starting on Thursday, Aug 22, and stop enrollment one week later on Thursday, Aug 29.

This study is similar to the recent studies run for the new Pocket experience on the new Tab (18 branch, 4 branch and 2 branch studies) to monitor impact on Firefox Engagement metrics and Search revenue.

For reference:
18 branch: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66layoutstest1/
4 branch: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66-confirmation-test/
2 branch: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-68-2-branch-confirmation-test-v2/

Comment 2

[Jessilyn Davis]

PI request here: https://jira.mozilla.com/browse/PI-249

Comment 3

[Jessilyn Davis]

Update on timing:

After talking with the experimenter team, it's recommended that holdback study enrollment start ON the day of Release, Sept 3, and complete enrollment a week later, Sept 10. Then we'd rollout the pref ON in Sept 10.

The good news: That gives us time for QA by 8/27, and more time to scope out this experiment before launch.

The not so great news: Anyone who has been in a past experiment (18 branch, 4 branch, 2 branches above), and upgrade to Fx 69 between Sept 3 - Sept 10, would see the old Activity Stream experience (1 row, 3 cards), until the pref is ON in Sept 10 to see the new experience. A large % of our en-US audience is currently in an experiment, granted a large % of experiment enrollers have been getting the control or basic layouts (1 row, 3 cards) vs. a new, Pocket layout.

I'm unsure if this is a blocker from a Product perspective (NI'ing Tawanda and Nick for awareness) - but this is our option for doing a holdback study.

Comment 4

[Nick Chapman]

Just recording here that we confirmed on Slack that Tawanda and I are comfortable from the product perspective with this user experience in order to do the holdback study.

Comment 5

[Jessilyn Davis]

Per email thread with Rebecca, Nick, Tawanda, Reese - Ben is taking the lead on this. Thank you, Ben!

Comment 6

[Jessilyn Davis]

For context - our Pocket + New Tab: Shipping in 69 presentation: https://docs.google.com/presentation/d/1RddQWwIgyWvrhEsxrP-8EcKM_HYxOXW1dvGv5zstcfY/edit#slide=id.p

Comment 7

[Ben Miroglio [:bmiroglio]]

report is here:
https://mozilla-private.report/newtab-spoc-experiment/index.html