1557131 - [Shield] Pref-Flip: activity-stream-pktnewtabH12019-release67-2-branch-confirmation-test-1557131

Status: NEW

(Shield :: Shield Study, task, P3)

Description

[Experimenter Bot]

su:tawanda:placeholder

Pocket Newtab 2 branch confirmation study to determine impact on search volume.

More information: https://experimenter.services.mozilla.com/experiments/sutawandaplaceholder/

Comment 1

[Su-Young Hong]

[Tracking Requested - why for this release]:

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

[Tracking Requested - why for this release]:

Comment 3

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

Hey everyone, we've found a couple of inconsistencies in the available documentation, the experimenter and the actual results that we're looking for.

The value of the variant branch from the Study Documentation gdoc is {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":false,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=3-col-7-row-octr.json"} which has the enabled parameter set to false and the value of the variant branch from the experimenter page is {"enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=3-col-7-row-octr.json"}.

Regardless of which pref we use, the New Tab page does not populate with the expected results. In Nightly we get the "Oops! We almost loaded this section, but not quite." message and in Release the page only has the Search bar displayed.

In the meantime we've found that this value {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=3-col-7-row-octr"} works as expected, but we are unsure if this is the correct value that we should be using.

@tim, @tkanhema, could you please look over the documentation and update the values?

Comment 4

[Jessilyn Davis]

@Su - can you look too? I think this was something you played with when showing me/Tawanda how to set up Experimenter. (IIRC, we had to play with making the branches different to save the placeholder setup, and I remember thinking "i wonder if we'll need to remember to go back in there and change the false/true piece.")

Comment 5

[Scott [:thecount] Downe]

In nightly using {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":false,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=3-col-7-row-octr.json"} I get a page with Discovery Stream turned off, because that has it set to enabled: false

For the other one, "https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=3-col-7-row-octr.json" isn't working, dunno why, returns a 400.

Comment 6

[Scott [:thecount] Downe]

This seems to work though: https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=3-col-7-row-octr

Seems there shouldn't be a .json at the end of that.

Comment 7

[Tawanda Kanhema]

Test values have now been updated in Experimenter and in the test documentation. Please let us know if you need any more information. https://experimenter.services.mozilla.com/experiments/sutawandaplaceholder/

Comment 8

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

Thanks everyone, I think it's going to be smooth sailing from now on.

Comment 9

[Experimenter Bot]

    Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment 3-col-7-row-octr 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=3-col-7-row-octr"}

3 Columns 7 Rows powered by OCTR
        
- Treatment control 50%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":false,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}

normal behavior
        

    What version and channel do you intend to ship to?

20% of Release Firefox 67.0

    Are there specific criteria for participants?

Additional filters:
Prefs: 

browser.newtabpage.enable
OR (we want AT LEAST one of these to be default)
browser.startup.homepage

Enroll profiles that are on: 
•     - FF66 (this version supports our experience and we want to include users on older versions)
•     - FF67
•     - FF68 (so users don't get kicked out of the experiment when they upgrade)

AND (we want ALL of the below to be default)

browser.newtabpage.activity-stream.showSearch
AND
browser.newtabpage.activity-stream.feeds.topsites
AND
browser.newtabpage.activity-stream.feeds.section.topstories
AND
browser.newtabpage.activity-stream.feeds.section.highlights

Exclude the Following:  

Please exclude all subjects that were enrolled (at any point) in the following experiments:
* https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66layoutstest1/
* https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtabh12019-release66-confirmation-test/
* (upcoming) https://experimenter.services.mozilla.com/experiments/activity-stream-pktnewtab-1555451-release67syndicatedboost/#population-comments

Any additional filters:

Please target users on FF66 and above (so FF67, FF68, etc.).

Please exclude users on Dark Theme:
['environment']['addons']['activeAddons']['theme']['id'] = '[e-mail]'

AddOn Exclusions:

    What is your intended go live date and how long will the experiment run?

Jun 25, 2019 - Sep 23, 2019 (90 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

* We are looking to measure at least 1% change in average total search (over the observation period). 

Do you plan on surveying users at the end of the experiment? Yes/No.
Strategy and Insights can help create surveys if needed

N/A

    Who is the owner of the data analysis for this experiment?

[e-mail]

    Will this experiment require uplift?

False

    QA Status of your code:

not started

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/sutawandaplaceholder/

Comment 10

[Su-Young Hong]

Update: Experiment broken

Due to a default pref value change from FF68, control users are being un-enrolled from this experiment.

More details here.

The data from this experiment is no longer valid (post July 8th - we have 1 week of usable data from this experiment)

We will need to re-run if we want valid results.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Per Experimenter, this ended on July 16.