1617643 - CFR Experiment for New Users to be prompted to import passwords

Status: RESOLVED INACTIVE

(Data Science :: Experiment Collaboration, task)

Description

[Tim Spurway [:tspurway]]

Brief Description of the request (required):

If we offer new desktop browser users the opportunity to import their passwords when visiting a site that requires logins, then we will see less users returning to their former browser because we have observed that missing features are a push force on users. The hypothesis is that users who are given the opportunity to import passwords from the browser they used to install Firefox will keep using Firefox.

Business purpose for this request (required):

This experiment seeks to improve 7 day retention for new users.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required): Dave Camp explicitly asked for an experiment to start before Feb 27, 2020

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

https://docs.google.com/document/d/1WzaoVFERVtwizsvl01yqsc4BQA3I_Jk6tg4h8Vt-al4/edit#

Name of Data Scientist (If Applicable): Su

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Nan Jiang [:nanj]]

Hi Ciprian,

We're looking to run this CFR message as an experiment in Firefox 73 ASAP. The message is ready and could be published to stage for testing now, wonder what's the best way to get the QA sign-off for this one in a timely manner. Could you shed a light on this, please?

Thanks!

Comment 2

[Ron Manning]

Here is the link to the PI Request for the Shield study: https://jira.mozilla.com/browse/PI-503

Comment 3

[Ciprian Muresan [:cmuresan], Ecosystem QA]

Hey Nan,

Thanks for the ping! We should go ahead and publish the message to Remote Settings Stage as we will want to test this as closely as possible to how users will experience it. We will also be using a Normandy Recipe to target chrome and edge switchers while doing so.

As for being able to sign-off in a timely manner, I've seen that you updated the due date to the 28th of Feb, this should be totally doable on our side if the message is published ASAP. Also given that time is relatively short we will forgo test case creation for the request and test the CFR based on our prior experiences. We will sign-off by Friday EOD (our time) and should any issues appear we will ping you as fast as we can.

Let me know if this works for you/the team.

Comment 4

[Nan Jiang [:nanj]]

Hey Ciprian,

This message is now on stage.

Steps to test:

• This targets users whose profile age is 7 days younger for Firefox 73 (on Wondows + en-US). Note that like the Chrome switcher experiment, we will use Normandy to enroll new users if they download Firefox from Chrome and Edge (i.e. attribution.us in ["chrome", "edge"]). We don't check this in the CFR targeting assuming that the enrolled users should already qualify this condition.
• Use remote settings devtool to download the CFR message on stage, the ID for this message is CFR_EXP_IMPORT_PASSWORD
• Change the value of pref browser.newtabpage.activity-stream.asrouter.providers.cfr to be {"id":"cfr","enabled":true,"type":"remote-settings","bucket":"cfr","frequency":{"custom":[{"period":"daily","cap":1}]},"categories":["cfrAddons","cfrFeatures"],"updateCycleInMs":3600000,"cohort":"import_password_treatment"}. the cohort key indicates we're in the experiment cohort. In reality, Normandy will do this when it enrolls users to the experiment cohort
• Make sure your profile is younger than 7 days
• Visit any of sites such as "facebook.com", "twitter.com", "youtube.com", you should see a CFR recommendation to ask if you want to import passwords from other browsers
• For non-experiment users (i.e. the pref value doesn't match against the above value), visiting those sites should not trigger the recommendation at all

Notes:

• This CFR has a daily frequency cap as 1, so you should only see one recommendation per day
• This CFR doesn't check the login status, therefore you might still see the recommendation even if you've already logged in

Comment 5

[Marius Coman [:mcoman], Ecosystem QA]

CFR Shield Study for Import Password
Firefox Release 73

We have finished testing the [PI-503] CFR Shield Study for Import Password experiment.

Quality status: GREEN - SHIP IT

Why is this experiment green?

• We have not found any issues during testing.

Testing Summary:

• Verified that the “CFR_IMPORT_PASSWORD” recommendation is successfully triggered if the websites from the documentation are accessed.
• Verified that the message UI respects the specifications from the documentation.
• Verified that the message UI respects the browser’s “Dark” theme colors.
• Verified that the “Import Wizard” is successfully opened if the message’s “Import Passwords” button is clicked.
• Verified that the message is no longer displayed and appears as blocked in the “about:newtab#devtools” if the “Don’t Show Me This Recommendation” option is selected.
• Verified that the message is displayed only once per day and for a maximum amount of 3 times.
• Verified that the message is not displayed if the “Recommend features as you browse” option from the “about:preferences” page is unchecked.
• Verified that the “CFR_EXP_IMPORT_PASSWORD" related telemetry pings are successfully displayed in the Browser Console.
• Using a local telemetry server, verified that the related telemetry pings are successfully sent and received by the server.
• Verified that the message is not displayed on profiles older than 7 days.
• Verified that the message is not displayed on other Firefox locales except “en-US”.
• Verified that the users with Firefox profiles containing “chrome", "edge" attribution values can be successfully targeted using a Normandy recipe.

Tested Platforms:

• Windows 10 x64
• Windows 7 x64

Tested Firefox versions:

• Firefox Release 73.0 en-US

Comment 6

[Ron Manning]

The dates for the experiment have been changed:

March 10, 2020 - April 7, 2020, Total Duration: 28 Days

March 10, 2020 - March 17, 2020, Enrollment: 7 Days

March 17, 2020 - April 7, 2020, Observation: 21 Days
https://experimenter.services.mozilla.com/experiments/activity-stream-password-import-cfr/

Comment 7

[Nan Jiang [:nanj]]

Let's keep an eye on bug 1619981 as they both share the same Normandy targeting on attributions.

Comment 8

[Experimenter Bot]

[Experiment]activity-stream password import cfr status has been changed to: Live
            url:https://experimenter.services.mozilla.com/experiments/activity-stream-password-import-cfr/

Comment 9

[Jess Mokrzecki [:jmok]]

Work for the DS team is now tracked in Jira. You can search with the Data Science Jira project for the corresponding ticket.