1549583 - [Shield] Pref-Flip Study: activity-stream-pktnewtabH12019-release66-confirmation-test

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Experimenter Bot]

activity-stream-pktnewtabH12019-release66-confirmation-test

4 branch, large scale study to get high resolution tests on overall browser metrics for New Tab layout candidates (3 candidates + control) for releasing on FF68.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66-confirmation-test/

Comment 2

[Pascal Chevrel (relman team) -> :pascalc|PTO until Sept 2]

Jared, what versions of Firefox are affected by this study? 68 only?

Comment 3

[Jared Lockhart [:jlockhart]]

:pascalc According to the experiment definition it says: Please target users on FF66 and above (so FF67, FF68, etc.).

Comment 4

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the study and what values should
    each branch be set to?

- Treatment 3-col-1-row-octr-4-col-6-row-categories 25%:

Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"collapsible\":true,\"enabled\":true,\"show_spocs\":true,\"hardcoded_layout\":false,\"personalized\":false,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=3-col-1-row-octr-4-col-6-row-categories\"}"

3 column first row using the OCTR feed, followed by 6 rows of 4-column category feeds.
        
- Treatment 3-col-7-row-octr 25%:

Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"collapsible\":true,\"enabled\":true,\"show_spocs\":true,\"hardcoded_layout\":false,\"personalized\":false,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=3-col-7-row-octr\"}"

3x7 grid of cards using the OCTR feed and followed by Popular Topic links.
        
- Treatment basic-3-col-octr 25%:

Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"collapsible\":true,\"enabled\":true,\"show_spocs\":true,\"hardcoded_layout\":false,\"personalized\":false,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic-3-col-octr\"}"

3 stories from the OCTR feed with Popular Topic links below.
        
- Treatment control 25%:

Value: "{\"api_key_pref\":\"extensions.pocket.oAuthConsumerKey\",\"enabled\":false,\"show_spocs\":true,\"layout_endpoint\":\"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic\"}"

This is the default experience without Discovery Stream enabled.
        

    What version and channel do you intend to ship to?

5% of Release Firefox 66.0

    Are there specific criteria for participants?

Prefs:

browser.newtabpage.enable
        OR (we want AT LEAST one of these to be default) 
browser.startup.homepage

        
        AND (we want ALL of the below to be default)  

browser.newtabpage.activity-stream.showSearch
AND 
browser.newtabpage.activity-stream.feeds.topsites
AND
browser.newtabpage.activity-stream.feeds.section.topstories
AND 
browser.newtabpage.activity-stream.feeds.section.highlights

Studies:

Please exclude all subjects that were enrolled (at any point) in the following experiments: 
* https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66layoutstest1/
* https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66smoketest2/

Any additional filters:

Please target users on FF66 and above (so FF67, FF68, etc.). 

Please exclude users on Dark Theme:
    ['environment']['addons']['activeAddons']['theme']['id'] = '[e-mail]'

    What is your intended go live date and how long will the study run?

May 14, 2019 - Aug 06, 2019 (84 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

The success metrics are defined here: https://docs.google.com/document/d/1iwrKp3JT1WhULIlnv0r5ZahTrSdTSQ0RY2BrZv9_tss/edit?usp=sharing

Do you plan on surveying users at the end of the study? Yes/No.
Strategy and Insights can help create surveys if needed

Yes.

    Who is the owner of the data analysis for this study?

[e-mail]

    Will this experiment require uplift?

False

    QA Status of your code:

This experiment will be launching at the end of FF66, and will continue through the release of FF67. 

Please test over both versions. 

QA guidelines here: https://docs.google.com/document/d/1iySmNA2tBl3ApLuJtP7UBlPwNx9sH1SmK-jG6exvMQ0/edit#heading=h.s177b01wzbij

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66-confirmation-test/