1601326 - Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72

Status: RESOLVED INACTIVE

(Data Science :: Experiment Collaboration, task)

Description

[Tim Spurway [:tspurway]]

Brief Description of the request (required):

We would like to run a study in Firefox 72 to test whether users engage more often, or at a higher rate, with dynamic or static onboarding triplets. Our hypothesis is that users who do engage with the onboarding triplets will continue to discover new and interesting features by clicking on triplet cards on subsequent newtab pages. This behaviour would be indicated by higher click-through-rates for users who click on triplet cards.

In addition, we would expect that this behaviour would not annoy users. We wouldn’t expect to see a significant increase in the triplet section dismiss rate (the number of users closing the triplets section).

Lastly we would expect this feature to do “no-harm” as far as standard Firefox engagement and retention metrics.

Business purpose for this request (required):

This test is in support of Firefox growth initiatives.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Firefox 72

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

Meta Bug
Tech Docs
PRD

Name of Data Scientist (If Applicable):

Su

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Tim Spurway [:tspurway]]

Added Experimenter Ticket

Comment 2

[Su-Young Hong]

Note: this ticket is covering 2 experiments

• main experiment for release: https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72/
• beta experiment to test normandy first run delivery: https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-beta/

Comment 3

[Experimenter Bot]

[Experiment]activity-stream Contextual Onboarding Framework Experiment in BETA status has been changed to: Ship
            url:https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-beta/

Comment 4

[Su-Young Hong]

Beta experiment (for testing normandy deployment of feature) analysis has been done:

• Dynamic Triplets FF72 Beta Smoke Test Report

TLDR

No major issues that would block this specific experiment, but seeing some suspicious stuff that should be checked out going forward:

• FXA data is coming in, but it’s really low, could be explained by UTM tagging implementation.
  • Q: @Kate: Was the about:welcome page email form impression implemented with “trailhead-join-branch”?
• Only about 68% of profiles are hitting a modal/triplet impression. This seems very suspicious. Unclear exact reason.
• A good chunk of users are sending us untagged data (and it’s more skewed for Activity Stream data). Not clear if this is just a data tagging issue, or if it’s a feature activation issue.

Overall enrollment rate, branch assignment rate, and unenrollment rate look as we’d expect. No issues there.

For the subset of users who hit a branch-specific card, the branch assignments is correct (so the feature IS working, at least in some cases for some users).

Comment 5

[Experimenter Bot]

[Experiment]activity-stream Contextual Onboarding Experiment for Dynamic Triplets on Newtab in Firefox 72 V2 status has been changed to: Live
            url:https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-v2/

Comment 6

[Jess Mokrzecki [:jmok]]

Work for the DS team is now tracked in Jira. You can search with the Data Science Jira project for the corresponding ticket.