1529327 - [Shield] Pref-Flip Study: activity-stream-pkt-newtab-beta66complexandperformance2, 66, Beta

Status: RESOLVED FIXED

(Shield :: Shield Study, defect, P1)

Description

[Experimenter Bot]

activity-stream-pkt-newtab-beta66complexandperformance2

Summary: This is a simple 3 branch test to check to see if complex layouts in the discovery-stream API has impact on performance (which is measured with new activity stream telemetry), and to test out our new API and changes to the activity stream telemetry.

A continuation of: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66smoketest1/

Purpose: Check if there are major issues with usage and performance that arise with the complex layout using the discovery-stream API.

Branches:

• Control (current experience)
• basic (replicate current experience with new discovery-stream API)
• 66-beta-2-complex (This is a more complex layout with a hero element on top and multiple topic rows below.)

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66complexandperformance2/

Comment 1

[Su-Young Hong]

[Tracking Requested - why for this release]: experiment, following directions from: experimenter instructions: bugzilla updated

Comment 2

[Ciprian Muresan [:cmuresan], Ecosystem QA]

We have finished testing the activity-stream-pkt-newtab-beta66complexandperformance2 experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• We haven’t found any issues during testing.

Testing Summary:

• Testrail

Tested Platforms:

• Windows 10 x64
• Mac 10.14.2
• Arch Linux 4.14.3

Tested Firefox versions:

• Firefox Beta 66.0b9 & 66.0b10

Comment 3

[Jessilyn Davis]

Liz, can you give us the Relman greenlight by 9:30am PT so we can get this launched during the Experimenter review mtg at 10am PT?

Thanks! And thanks you, Ciprian!

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Yes, signed off in https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66complexandperformance2/.

Comment 5

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the study and what values should
    each branch be set to?

- Treatment control 33%:

Value: {"enabled":false,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=basic"}

Control branch, no changes to existing experience
        
- Treatment default 33%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}

Replication of control branch with new discovery-stream API
        
- Treatment complex 34%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=66-beta-2-complex"}

This is a more complex layout with a hero element on top and multiple topic rows below.
        

    What version and channel do you intend to ship to?

15% of Beta Firefox 66.0

    Are there specific criteria for participants?

Locales:
* en-US only

Geographic regions:
* US only

Prefs:
* please exclude any users with non-default values in the following prefs:
- `browser.newtabpage.enable`
- `browser.startup.homepage`
- `browser.newtabpage.activity-stream.showSearch`
- `browser.newtabpage.activity-stream.feeds.topsites`
- `browser.newtabpage.activity-stream.feeds.section.topstories`
- `browser.newtabpage.activity-stream.feeds.section.highlights`

Studies:
* please exclude any profiles that have been enrolled in: 
    - activity-stream-pkt-newtab-beta66smoketest1    
    - experiment link: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66smoketest1/ 

Any additional filters:
* please exclude any users that have the Mozilla Dark Theme activated: `['environment']['addons']['activeAddons']['theme']['id']` is `'[e-mail]'`
    

ADDITIONAL NOTE: the code for this experiment is not currently landed in Beta66 yet. There will be an uplift landing the code on Friday. There will be an additional filter to target only those users who have updated with the latest code (will update in comments)

    What is your intended go live date and how long will the study run?

Feb 25, 2019 - Mar 27, 2019 (30 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We are looking for non-regression in browser usage metrics (days used, retention, hours, active hours, uris, and searches). We are looking for increase in AS and pocket engagement metrics. We are looking for non-regressions in AS performance metrics. 

Do you plan on surveying users at the end of the study? Yes/No.
Strategy and Insights can help create surveys if needed
 
No.

    Who is the owner of the data analysis for this study?

[e-mail]

    Will this experiment require uplift?

False

    QA Status of your code:

The currently public version of Beta does not have all the code we need for this experiment. It will be uplifted on Friday, and users on Monday will have the code. 

However, for QA purposes, please follow the instructions provided here: 

https://docs.google.com/document/d/1iySmNA2tBl3ApLuJtP7UBlPwNx9sH1SmK-jG6exvMQ0/edit#heading=h.s177b01wzbij

Under the section: tinderbox build (relman has provided a tinderbox build with the changed uplifted before the official uplift date for us to QA on)

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66complexandperformance2/

Comment 6

[:shell escalante]

enrollment paused March 4th per Experimenter design

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Can we close this out now that the experiment ended?

Comment 8

[Su-Young Hong]

Hey liz,

Just to be clear, this experiment is still running, enrollment has just been paused (meaning no new subjects in experiment, but those already enrolled will continue).

Did you want to close the bug because launching/enrollment is complete? Or should we close this when the experiment has run it's course?

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

No, we should keep it open. I wanted to make sure there's nothing new to do here for 66 release. Thanks!

Comment 10

[Josh Gaunt [:jgaunt]]

Recipe 698 has been disabled - this experiment has concluded.