Closed Bug 1687532 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-au-nz-v2 Fx 83.0 to 87.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox85+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox85 + fixed

People

(Reporter: experimenter, Assigned: kdemtchouk)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Control control 50%:

Value: None

Australia and New Zealand do not see 7 rows of stories
        
- Treatment variant 50%:

Value: None

Australia and New Zealand see 7 rows of stories
        

    What version and channel do you intend to ship to?

100% of Release Firefox 83.0 to 87.0

    Are there specific criteria for participants?

Exclude anyone in the non-control - AKA the treatment -  variant of this test:
https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-au-nz/

Also:
common Pocket Filter 1
Countries: Australia (AU) New Zealand (NZ) 

Locales: English (British) (en-GB) English (Canadian) (en-CA) English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Feb 02, 2021 - Apr 06, 2021 (63 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We'll measure engagement with Pocket stories, as quantified by these metrics:

-Pocket recommendation click rate

-% of the test audience that clicked at least one recommendation during 4 weeks

-% of the test audience that disables Pocket recommendations

We will also monitor these Firefox metrics to make sure they do not decline for those users in the study:

-Search Count per User

-Ad Click Count per User

-Retention

-URI Count per User

-active days of use

For the Search Count and Retention metrics, specifically, we will also look at the results for the Regular and New & Resurrected user cohorts for all 4 weeks of the study.

Do you plan on surveying users at the end of the delivery? No. Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

Requested: https://jira.mozilla.com/browse/PI-926; 

QA Status: GREEN. For more information see https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-au-nz-v2/#comment1322.

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-au-nz-v2/

activity-stream-pocket-new-tab-au-nz-v2

Re-launch of study to display Pocket New Tab recommendations to en-* users in Australia and New Zealand.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-682
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-au-nz-v2/

[Tracking Requested - why for this release]:
Launching another AU/NZ experiment (take 2) - to the CONTROL branch of the first AU/NZ experiment:
https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-au-nz/

Launching on 2/2 (when the first experiment ends)

tracking-firefox85: --- → ?
status-firefox85: --- → affected
tracking-firefox85: ? → +
Whiteboard: [no-nag]

Pocket New Tab - en-* - Australia and New Zealand - Take 2 - 2/2
Firefox Release 83.0, 84.0.2, and 85.0

We have finished testing the [PI-926] Pocket New Tab - en- - Australia and New Zealand - Take 2 - 2/2* experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • Delivery mechanism:
    • Initially QA raised a concern regarding a possible conflict between unenrollments from the 1st experiment and enrollments in the 2nd experiment. While the conflict is still possible, the concern is alleviated by the fact that clients will successfully enroll in the 2nd experiment after the 1st one is marked as completed.
  • Product specific:
    • No issues found.

Testing Summary:

Tested Platforms:

  • Windows 10
  • macOS 11.1
  • Linux Mint 20

Tested Firefox versions:

  • Firefox Release 85.0, BuildID 20210118153634
  • Firefox Release 84.0.2, BuildID 20210105180113
  • Firefox Release 83.0, BuildID 20201112153044

Regards,
Ciprian Muresan
Ecosystem QA Team

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-au-nz-v2 → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-au-nz-v2 Fx 83.0 to 87.0 Release
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox85: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.