1524139 - [Shield] Pref-Flip Study: activity-stream-pkt-newtab-beta66smoketest1

Status: NEW

(Shield :: Shield Study, defect)

Description

[Experimenter Bot]

activity-stream-pkt-newtab-beta66smoketest1

Summary: This is a simple smoke test in beta66 to check if the discovery-stream API is working properly and nothing is totally broken. This experiment is meant to be a trial run for the new experiments process and internal team processes.

Purpose: Check if any massive issues with usage, data reporting, and work out any kinks in the team's experiment launching process.

Branches:

• Control (current experience)
• basic (replicate current experience with new discovery-stream API)

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66smoketest1/

Comment 1

[Ciprian Muresan [:cmuresan], Ecosystem QA]

We have finished testing the activity-stream-pkt-newtab-beta66smoketest1 experiment.

We have found 2 new issues during testing:

• 1524294 - [Experiment] The Discovery Stream page is not responsive when horizontally resized
• 1524295 - [Experiment] Spocs are not dismissed/replaced after a page refresh

QA’s recommendation: YELLOW - SHIP IT CONDITIONALLY

Reasoning:

• We are concerned especially about bug 1524295 due to the fact that sponsored content is not dismissible as in the default experience. We are still unsure if this issue is a blocking one or not. As it stands now, the team acknowledged that it is a valid issue.
• We are mostly concerned that this could lead to a negative impact in the user's experience during this experiment, especially the ones that see the sponsored content as ads.

Testing Summary:

• Verified that the layout is changed to mimic the default experience.
• Verified that the Search bar works as expected.
• Verified that the applicable scenarios on the Top Sites section (adding, editing, removing, pinning/ unpinning, dragging and dropping).
• Verified that all links correctly redirect to their appropriate pages.
• Verified that the user can opt-out of the experiment from the about:preferences#home (only applicable if the experiment is served by Normandy) and that the user is successfully reverted to the default experience.

Tested Platforms:

• Windows 10 x64
• MacOS 10.14.2
• Arch Linux 4.14.3 x64

Tested Firefox versions:

• Firefox 66.0b3

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

From talking with Jess, sounds like this is ready to launch on Monday. Please consider this comment as relman signoff for beta 66 experiment.

Comment 3

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the study and what values should
    each branch be set to?

- Treatment control 50%:

Value: {"enabled":false,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=basic"}

The current newtab/homepage experience. 

PLEASE KEEP THIS PREF AS DEFAULT
        
- Treatment basic 50%:

Value: {"enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=basic"}

Replication of the current newtab/homepage experience, using the discovery-stream API. 

Should be the same as current newtab/homepage except highlights section is missing
        

    What version and channel do you intend to ship to?

1% of Beta Firefox 66.0

    Are there specific criteria for participants?

Locales: 
* en-US only

Geographic regions: 
* US only

Prefs: 
* please exclude any users with non-default values in the following prefs: 
	- `browser.newtabpage.enable`
	- `browser.startup.homepage`
	- `browser.newtabpage.activity-stream.showSearch`
	- `browser.newtabpage.activity-stream.feeds.topsites`	
	- `browser.newtabpage.activity-stream.feeds.section.topstories`
	- `browser.newtabpage.activity-stream.feeds.section.highlights`

Studies: 
* N/A

Any additional filters: 
* please exclude any users that have the Mozilla Dark Theme activated: `['environment']['addons']['activeAddons']['theme']['id']` is  `'[e-mail]'`

    What is your intended go live date and how long will the study run?

Feb 04, 2019 - Mar 18, 2019 (42 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

* The main effect we are looking for is any major breakages in our new API. We'll be looking for LARGE (obvious) differences in usage (days used, total session hours, total active hours, total searches, and total URI), as well as any activity-stream, pocket, and performance specific metrics. 

* Again, we are dry running our API before launching experiments testing different layouts of newtab. 

* This experiment is to make sure we can launch experiments and collect data as expected, and to pick up any unexpected data issues before launching experiments testing different layouts. 

Do you plan on surveying users at the end of the study? Yes/No.
Strategy and Insights can help create surveys if needed

* No.

EXTRA: study sizing. We're not looking for any specific effects from this study other then confirm the API is working in the pre-release population.

WAU of Beta for en-US, US estimated at about 100K (https://sql.telemetry.mozilla.org/queries/61331/source). We expect maximum of 80% of them to fulfill all of our targeting criteria (from eyeballing). Will be important data to find out how many subjects 1% sample of WAU produces (with these targeting specs).

    Who is the owner of the data analysis for this study?

[e-mail]

    Will this experiment require uplift?

False

    QA Status of your code:

We have internal QA doing additional testing on this pref layout. Link to follow.

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66smoketest1/

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Untracking since we have this information handy in Experimenter.