Closed Bug 1615344 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: Chrome Switchers card for Onboarding Triplets in Firefox 73 Fx 73.0 to 74.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox73+ fixed, firefox74+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox73 + fixed
firefox74 + fixed

People

(Reporter: experimenter, Assigned: aoprea)

References

(
URL
)

Details

(Whiteboard: [no-nag])

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

trailhead.firstrun.branches

    What are the branches of the experiment and what values should
    each branch be set to?

- Control join-dynamic 50%:

Value: join-dynamic

cards are dynamic cards, minus chrome card
        
- Treatment join-dynamic_chrome 50%:

Value: join-dynamic_chrome

cards are dynamic cards, including chrome card
        

    What version and channel do you intend to ship to?

25% of Release Firefox 73.0 to 74.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:

Targeting should be for new profiles only (profile creation date after start of experiment date
PLEASE MAKE STICKY: trailhead.firstrun.didSeeAboutWelcome is false

Additionally, targeting should be for Chrome attributed users only (environment.settings.attribution.ua = 'chrome')
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Feb 20, 2020 - Apr 02, 2020 (42 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

* Overall usage metrics (retention, days used, URIs, activehours, searches) 
* % of users who clicked chrome tile (variant only) 
* % of users who started import
* % of users who completed import

Supplementary metrics to check: 
* jank during import process

Do you plan on surveying users at the end of the delivery? Yes/No.
Strategy and Insights can help create surveys if needed

N/A

    Who is the owner of the data analysis for this experiment?

shong@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Sent a GREEN sign-off as per comment: https://experimenter.services.mozilla.com/experiments/chrome-switchers-card-for-onboarding-triplets-in-firefox-73/#comment652

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/chrome-switchers-card-for-onboarding-triplets-in-firefox-73/

Chrome Switchers card for Onboarding Triplets in Firefox 73

We would like to run a study targeting Windows only Firefox 73 users that downloaded Firefox using Chrome.

We would show part of these users a Chrome specific language card (as of of the 3 onboarding cards) inviting them to import their data from Chrome.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/chrome-switchers-card-for-onboarding-triplets-in-firefox-73/

Chrome Switcher Experiments

Firefox Release 73

We have finished testing the [PI-352] Chrome Switcher Experiments experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • We have found one new issue during testing (Bug 1615635) and encountered one known enhancement (Bug 1598597). Both the new issue and the known enhancement encountered during testing do not affect the experiment end users in a harmful way.

  • We have not managed to perform end-to-end testing in the true meaning of it because in order to do that we would have needed to use the Normandy Prod server and when doing that we’ve hit a Normandy specific issue which blocked us from further testing. Instead, we used the Normandy stage server to deliver the recipe, created a new profile right after Firefox was installed using the stub installer (previously downloaded with Chrome) and with the help of a pref.js file made all the necessary pref changes to be targeted by Normandy. This scenario is much slower to load our pref change than the Normandy Prod coupled with the stub installer, but we did manage to see the cards we needed without any delay.

Testing Summary:

Tested Platforms:

  • Windows 10 x64

  • Windows 7 x64

Tested Firefox versions:

  • Firefox Release 73.0 en-US

  • Firefox Release 73.0 es-ES

  • Firefox Release 73.0 fr

  • Firefox Release 73.0 ar

[Tracking Requested - why for this release]: This is an experiment looking to run on release and the following firefox version.

status-firefox73: --- → verified
status-firefox74: --- → verified
tracking-firefox73: --- → ?
tracking-firefox74: --- → ?
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Chrome Switchers card for Onboarding Triplets in Firefox 73 → [Experiment] Pref-Flip Experiment: Chrome Switchers card for Onboarding Triplets in Firefox 73 Fx 73.0 to 74.0 Release
Start Date: 2020-02-20 End Date: 2020-04-02
Whiteboard: [no-nag]
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.