Closed Bug 1601763 Opened 6 years ago Closed 6 years ago

[Experiment] Pref-Flip Experiment: activity-stream Contextual Onboarding Framework Experiment in BETA Fx 72.0 to 73.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox72+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox72 + fixed

People

(Reporter: experimenter, Assigned: shell)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

set trailhead.firstrun.branches

    What are the branches of the experiment and what values should
    each branch be set to?

- Control static triplets 50%:

Value: join-static

this is the existing, static onboarding triplets experience
        
- Treatment dynamic triplets 50%:

Value: join-dynamic

this is the triplets that change over time
        

    What version and channel do you intend to ship to?

50% of Beta Firefox 72.0 to 73.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Only enroll new users - we get this by excluding users with old profiles (profile creations date is less than launch date.)  This is because we need telemetry date for this normally - but telemetry isn't guaranteed available that early.   We expect this will cover the pave-over scenario - install with old profile with new firefox - this filter exclude the pave-overs.  

Sticky:  Has not seen about:welcome page.  Exclude users who have seen the first run experience -  trailhead.firstrun.didSeeAboutWelcome  is true
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Dec 10, 2019 - Dec 17, 2019 (7 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We expect beta population to have few users who qualify - but those that do - we expect them to get enrolled in a branch and see the right treatment. 

We expect to not see time-outs.

    Who is the owner of the data analysis for this experiment?

shong@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

https://jira.mozilla.com/browse/PI-404
We don't want to cause issues with Ciprian's testing for the actual feature.  We believe they are ready to send out preliminary sign-off for the feature itself.  Mythmon can work with QA to avoid impacting other testing.

Testing this will be a dry run of the real thing shipping in January.  So passing this testing I think will mean passing testing for this experiment https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72/

QA Status: GREEN as per comment: https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-beta/#comment556

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-beta/

activity-stream Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72 BETA

Test Contextual Onboarding "triplet" cards in Firefox 72.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-beta/

[Tracking Requested - why for this release]: because I believe I have to set this flag for any experiment - even Beta. This is just testing the framework for an experiment we want to run in release 72 - to de-risk since it is the first time with this particular layout.

tracking-firefox72: --- → ?

Onboarding framework BETA experiment
Firefox Beta 72.0bx

We have finished testing the Onboarding framework BETA experiment.

QA’s recommendation: GREEN

Reasoning:

Testing Summary:

  • TestRail link
  • We have also created a Normandy recipe on the Stage environment and verified that clients are enrolled in the “join-static” and “join-dynamic” branches.

Tested Platforms:

  • Windows 10 x64
  • Mac 10.14
  • Debian 9

Tested Firefox versions:

  • Firefox Beta 72.0b3
  • Firefox Beta 72.0b4

Regards,
Patriciu

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: activity-stream Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72 BETA → [Experiment] Pref-Flip Experiment: activity-stream Contextual Onboarding Framework Experiment in BETA Fx 72.0 to 73.0 Beta
Start Date: 2019-12-10 End Date: 2019-12-17
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
status-firefox72: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.