Closed Bug 1680555 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-logged-out-save Fx 84.0 to 84.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox84+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox84 + fixed

People

(Reporter: experimenter, Assigned: kdemtchouk)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment variant_c 25%:

Value: None

variant c experience for logged out users clicking on the save to Pocket button.
        
- Treatment variant_b 25%:

Value: None

variant b experience for logged out users clicking on the save to Pocket button.
        
- Treatment variant_a 25%:

Value: None

variant a experience for logged out users clicking on the save to Pocket button.
        
- Control control 25%:

Value: None

Default experience for logged out users clicking on the save to Pocket button.
        

    What version and channel do you intend to ship to?

100% of Release Firefox 84.0 to 84.0

    Are there specific criteria for participants?

extensions.pocket.enabled = true
Countries: all

Locales: English (British) (en-GB) English (Canadian) (en-CA) English (US) (en-US) 

    What is your intended go live date and how long will the experiment run?

Dec 15, 2020 - Feb 09, 2021 (56 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

Our primary success metric will be the conversion rate for users that land on getpocket.com (after having clicked one of the signup buttons in the menu variants) and then convert to creating a Pocket account.

As supplementary metrics, we will also be comparing the click rates on the signup buttons (and other menu links) across each variant.

We will also monitor these Firefox metrics to make sure they do not decline for those users in the study:

-Search Count per User

-Ad Click Count per User

-Retention

-URI Count per User

For the Search Count and Retention metrics, specifically, we will also look at the results for the Regular and New & Resurrected user cohorts for all 4 weeks of the study.

Do you plan on surveying users at the end of the delivery? No. Strategy and Insights can help create surveys if needed

    Who is the owner of the data analysis for this experiment?

kirill@getpocket.com

    Will this experiment require uplift?

False

    QA Status of your code:

Not started. Requested: https://jira.mozilla.com/browse/PI-874

QA Status: GREEN. For more information, see https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-logged-out-door-hanger/#comment1252

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-logged-out-door-hanger/

activity-stream-pocket-new-tab-logged-out-save

Study to test Pocket door hanger menu styles on logged-out users.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DS-682
More information: https://experimenter.services.mozilla.com/experiments/activity-stream-pocket-new-tab-logged-out-door-hanger/

[Tracking Requested - why for this release]:
aiming to launch experiment on 12/15 w/ Fx 84 (where the feature functionality is :) )

tracking-firefox84: --- → ?

Logged out Pocket Save button doorhanger

Firefox Release 84

We have finished testing the [PI-874] Logged out Pocket Save button doorhanger experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • Delivery mechanism:
    • No issues found.
  • Product specific:
    • The new issue we have encountered on the feature part does not affect user experience in a harmful way.

Testing Summary:

Tested Platforms:

  • Windows 10 x64
  • Mac 10.15.6
  • Linux Mint 20 x64

Tested Firefox versions:

  • Firefox RC 84.0 en-US - Build ID: 20201207203640
  • Firefox RC 84.0 en-GB - Build ID: 20201207203640
  • Firefox RC 84.0 en-CA - Build ID: 20201207203640

Regards,

Marius Coman

Ecosystem QA Team

Whiteboard: [no-nag]
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream-pocket-new-tab-logged-out-save → [Experiment] Pref-Flip Experiment: activity-stream-pocket-new-tab-logged-out-save Fx 84.0 to 84.0 Release
User Story: (updated)
Start Date: 2020-12-15 End Date: 2021-02-09
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox84: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.