Closed Bug 1610962 Opened 6 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: activity-stream Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72 V2 Fx 72.0 to 73.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: shell)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

set trailhead.firstrun.branches

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment dynamic triplets 50%:

Value: join-dynamic

this is the triplets that change over time
        
- Control static triplets 50%:

Value: join-static

this is the existing, static onboarding triplets experience
        

    What version and channel do you intend to ship to?

25% of Release Firefox 72.0 to 73.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Only enroll new users - we get this by excluding users with old profiles (profile creations date is less than launch date.)  This is because we need telemetry date for this normally - but telemetry isn't guaranteed available that early.   We expect this will cover the pave-over scenario - install with old profile with new firefox - this filter exclude the pave-overs.  

Sticky:  Has not seen about:welcome page.  The recipe is excluding users who have  trailhead.firstrun.didSeeAboutWelcome  is true.  new users on first startup may have this not set at normandy execution, or have it set to false, both of which will make them eligible for the experiment.  non-new users will have this set to true.  when they start the browser and normandy executes, the value for this pref should exist for normandy to filter them out of the experiment.
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Jan 22, 2020 - Feb 26, 2020 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

Users who engage more with dynamic triplets would be indicated by higher click-through-rates for the triplet cards compared to control.

In addition, we would expect that dynamic triplets would not annoy users.  We wouldn’t expect to see a significant increase in the triplet section dismiss rate (the number of users closing the triplets section).

Lastly we would expect this feature to do “no-harm” as far as standard Firefox engagement and retention metrics.

    Who is the owner of the data analysis for this experiment?

shong@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

This was signed off as GREEN. For more information please see the following link:
https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72/#comment579

https://jira.mozilla.com/browse/PI-353
Believe this testing was covered in https://jira.mozilla.com/browse/PI-404

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-v2/

activity-stream Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72 V2

Test Contextual Onboarding "triplet" cards in Firefox 72.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-v2/

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72 V2 → [Experiment] Pref-Flip Experiment: activity-stream Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72 V2 Fx 72.0 to 73.0 Release
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.