[Shield] Pref-Flip Study: activity-stream-pkt-newtab-beta66complexandperformance2, 66, Beta

RESOLVED FIXED

Status

defect
P1
normal
RESOLVED FIXED
4 months ago
3 months ago

People

(Reporter: experimenter, Assigned: shong)

Tracking

unspecified
Dependency tree / graph
Bug Flags:
shield-qa +

Firefox Tracking Flags

(firefox66+ fixed, firefox67 affected)

Details

Reporter

Description

4 months ago

activity-stream-pkt-newtab-beta66complexandperformance2

Summary: This is a simple 3 branch test to check to see if complex layouts in the discovery-stream API has impact on performance (which is measured with new activity stream telemetry), and to test out our new API and changes to the activity stream telemetry.

A continuation of: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66smoketest1/

Purpose: Check if there are major issues with usage and performance that arise with the complex layout using the discovery-stream API.

Branches:

  • Control (current experience)
  • basic (replicate current experience with new discovery-stream API)
  • 66-beta-2-complex (This is a more complex layout with a hero element on top and multiple topic rows below.)

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66complexandperformance2/

Assignee

Comment 1

4 months ago

[Tracking Requested - why for this release]: experiment, following directions from: experimenter instructions: bugzilla updated

Blocks: 1512725
Summary: [Shield] Pref-Flip Study: activity-stream-pkt-newtab-beta66complexandperformance2 → [Shield] Pref-Flip Study: activity-stream-pkt-newtab-beta66complexandperformance2, 66, Beta
QA Contact: ciprian.muresan

We have finished testing the activity-stream-pkt-newtab-beta66complexandperformance2 experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found any issues during testing.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Mac 10.14.2
  • Arch Linux 4.14.3

Tested Firefox versions:

  • Firefox Beta 66.0b9 & 66.0b10
Flags: shield-qa+

Comment 3

4 months ago

Liz, can you give us the Relman greenlight by 9:30am PT so we can get this launched during the Experimenter review mtg at 10am PT?

Thanks! And thanks you, Ciprian!

Flags: needinfo?(lhenry)
Reporter

Comment 5

4 months ago
    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.newtabpage.activity-stream.discoverystream.config

    What are the branches of the study and what values should
    each branch be set to?

- Treatment control 33%:

Value: {"enabled":false,"show_spocs":true,"layout_endpoint":"https://getpocket.com/v3/newtab/layout?version=1&consumer_key=40249-e88c401e1b1f2242d9e441c4&layout_variant=basic"}

Control branch, no changes to existing experience
        
- Treatment default 33%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=basic"}

Replication of control branch with new discovery-stream API
        
- Treatment complex 34%:

Value: {"api_key_pref":"extensions.pocket.oAuthConsumerKey","enabled":true,"show_spocs":true,"layout_endpoint":"https://getpocket.cdn.mozilla.net/v3/newtab/layout?version=1&consumer_key=$apiKey&layout_variant=66-beta-2-complex"}

This is a more complex layout with a hero element on top and multiple topic rows below.
        

    What version and channel do you intend to ship to?

15% of Beta Firefox 66.0

    Are there specific criteria for participants?

Locales:
* en-US only

Geographic regions:
* US only

Prefs:
* please exclude any users with non-default values in the following prefs:
- `browser.newtabpage.enable`
- `browser.startup.homepage`
- `browser.newtabpage.activity-stream.showSearch`
- `browser.newtabpage.activity-stream.feeds.topsites`
- `browser.newtabpage.activity-stream.feeds.section.topstories`
- `browser.newtabpage.activity-stream.feeds.section.highlights`

Studies:
* please exclude any profiles that have been enrolled in: 
    - activity-stream-pkt-newtab-beta66smoketest1    
    - experiment link: https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66smoketest1/ 

Any additional filters:
* please exclude any users that have the Mozilla Dark Theme activated: `['environment']['addons']['activeAddons']['theme']['id']` is `'firefox-compact-dark@mozilla.org'`
    

ADDITIONAL NOTE: the code for this experiment is not currently landed in Beta66 yet. There will be an uplift landing the code on Friday. There will be an additional filter to target only those users who have updated with the latest code (will update in comments)

    What is your intended go live date and how long will the study run?

Feb 25, 2019 - Mar 27, 2019 (30 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

We are looking for non-regression in browser usage metrics (days used, retention, hours, active hours, uris, and searches). We are looking for increase in AS and pocket engagement metrics. We are looking for non-regressions in AS performance metrics. 

Do you plan on surveying users at the end of the study? Yes/No.
Strategy and Insights can help create surveys if needed
 
No.

    Who is the owner of the data analysis for this study?

shong@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

The currently public version of Beta does not have all the code we need for this experiment. It will be uplifted on Friday, and users on Monday will have the code. 

However, for QA purposes, please follow the instructions provided here: 

https://docs.google.com/document/d/1iySmNA2tBl3ApLuJtP7UBlPwNx9sH1SmK-jG6exvMQ0/edit#heading=h.s177b01wzbij

Under the section: tinderbox build (relman has provided a tinderbox build with the changed uplifted before the official uplift date for us to QA on)

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/activity-stream-pkt-newtab-beta66complexandperformance2/
Priority: -- → P1

Comment 6

4 months ago

enrollment paused March 4th per Experimenter design

Can we close this out now that the experiment ended?

Flags: needinfo?(shong)
Assignee

Comment 8

3 months ago

Hey liz,

Just to be clear, this experiment is still running, enrollment has just been paused (meaning no new subjects in experiment, but those already enrolled will continue).

Did you want to close the bug because launching/enrollment is complete? Or should we close this when the experiment has run it's course?

Flags: needinfo?(shong) → needinfo?(lhenry)

No, we should keep it open. I wanted to make sure there's nothing new to do here for 66 release. Thanks!

Flags: needinfo?(lhenry)
No longer blocks: 1512725

Recipe 698 has been disabled - this experiment has concluded.

Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.