1548072 - Trailhead First Run / Onboarding "Interrupts" experiment

Status: RESOLVED FIXED

(Data Science :: Experiment Collaboration, task, P1)

Description

[Tim Spurway [:tspurway]]

Brief Description of the request (required):

This experiment is to investigate various combinations of Trailhead messaging and UX flows in order to better understand how they affect Firefox engagement, short-term retention, search volume, and FxA usage.

Business purpose for this request (required):

This experiment is to support optimizing the effectiveness of the Trailhead initiative.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Trailhead launches on June 4th, 2019 with a new first run / onboarding flow and messaging. This experiment will launch simultaneously to a small percentage of the user population.

Note that because of the nature (First Run) of the experiment, it will be impossible to use Normandy to control this experiment. We will be doing cohort selection, enrolment, and telemetry markup manually within the feature code. The analysis of the telemetry, however, should be almost identical to any other experiment that is controlled by Normandy.

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

PRD: https://docs.google.com/document/d/1OaZWbRSl5jtXWatxfkPpAuhKK7Kkep5KAm_OtfNJogE/edit

Tech Docs: https://docs.google.com/document/d/1t3dDeaAKwLpH39a2LZ7xYzX4n17w-IGiTUpk0xU9LdY/edit

Comps: https://mozilla.invisionapp.com/share/ZFRRK6ZSQW9#/screens/360574173

Name of Data Scientist (If Applicable): Felix Lawrence

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Tim Spurway [:tspurway]]

Sizing / Population

We would want to run this experiment on the smallest population that would give us reasonable data to draw conclusions about variant effectiveness. Because the experiment will launch on June 4, 2019, there will be about 1 month before the release of Firefox 68, so that may affect the sizing. We will be primarily interested in comparing the following categories of metrics in each cohort:

• search volume per user from Firefox SAPs
• short-term user retention
• FxA engagement and usage
• 'classic' Firefox Telemetry engagement metrics

Variants

Control

The control group will be the existing (Firefox v. <= 67.0) first run / onboarding experience, and should be allocated 25% of the experiment population.

Direct to New Tab

This variant will remove all special UX and messaging in the Firefox first run flow and is detailed in bug #1548052. This cohort should be 25% of the experiment

Sync CTA messaging in Trailhead UX flow

This variant will present the current (Firefox v <= 67.0) onboarding messaging in the new Trailhead UX flow and is detailed in bug #1548065

Trailhead Messaging and Treatment

This variant is the actual Trailhead messaging and UX that will be released to non-experiment users in Firefox 67.0.5 and is described in the meta bug #1545802

Comment 2

[Felix Lawrence]

Is this a duplicate of 1545846? Shall we close it as a duplicate of this more detailed bug?

Or if this is the second experiment alluded to in yesterday's meeting, then we need to give the two experiments separate names and make it clear which is which.

Comment 4

[Felix Lawrence]

Power analysis

Comment 5

[Felix Lawrence]

The power analysis linked above recorded our options and not the decision: we decided to enrol 25% of new profiles for 4 weeks.

Comment 6

[Kate Hudson :k88hudson]

Hey would you be willing to help take another look at about:welcome in 67.0.5 just to make sure the experiment-related enrolment and Telemetry is working as expected / give this an R+ if everything looks good? (Code landed in https://github.com/mozilla/activity-stream/pull/4980, I can help with additional steps beyond what's in the tech doc if needed)

Comment 7

[Michael Cooper [:mythmon]]

The code in PR 4980 looks good. It matches how Normandy does the analogous telemetry. r+

Comment 8

[Felix Lawrence]

First interim report

Comment 9

[Felix Lawrence]

Second interim report

Comment 10

[Felix Lawrence]

Final report, reviewed today by :loines.