Closed Bug 1405822 Opened 8 years ago Closed 7 years ago

/firstrun email first test for fxa

Categories

(www.mozilla.org :: General, enhancement)

Product:
www.mozilla.org
Component:
General
Version:
Production
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: adavis, Assigned: agibson)

Details

(Whiteboard: [fxa])

Attachments

(2 files)

iframe.png
427.23 KB, image/png
Details
Link to GitHub pull-request: https://github.com/mozilla/bedrock/pull/5437
44 bytes, text/x-github-pull-request
Details | Review
Testing asking for email first has been a long time planned test. I've prepared a test recipe here: https://docs.google.com/document/d/1seUQEXOR-JGD6ff77iAP8lzDjW-BPengXugUzo-p2UQ/edit# The recipe is missing screenshots but below is the summary of the changes.. COPY Since the current copy says to login and that becomes irrelevant when we just ask for email, we propose a very simple copy change for the test to: >Take Firefox with You >Get your bookmarks, history, passwords, and other settings on all your devices. TRACKING To track the full FxA funnel, please past different UTM params to our iframe: -For the control, I propose utm_source=firstrun-login -For the treatment, I propose utm_source=firstrun-emailfirst iFRAME: I'm getting the details about what needs to change so that we can display the email first flow instead.
IIUC, in order to load email first instead of current iframe you would need to load /?action=email rather than /signin
Attached image iframe.png —
I believe this is what Alex is asking for. Just basic copy tweak on the left, and the Email First design on the right. See attached. I would much prefer a more ambitious goal of moving away from the iframed-login page. It's a bad practice, and tries to split the users’ attention. My guess is that few read what's on the left before signing up (hence why so many don't know what Sync does). Ideally this page would focus on selling Sync and subsequent steps could focus on set-up (not explanation of Sync). Here are three ideas that do not involve the iframe: https://youtu.be/rHrFCnuSJV0
(In reply to Ryan Feeley [:rfeeley] from comment #2) > Created attachment 8915343 [details] > iframe.png > > I believe this is what Alex is asking for. Just basic copy tweak on the > left, and the Email First design on the right. Perfect this is exactly it! Thank you > > I would much prefer a more ambitious goal of moving away from the > iframed-login page. It's a bad practice, and tries to split the users’ > attention. My guess is that few read what's on the left before signing up > (hence why so many don't know what Sync does). Ideally this page would focus > on selling Sync and subsequent steps could focus on set-up (not explanation > of Sync). > > Here are three ideas that do not involve the iframe: > https://youtu.be/rHrFCnuSJV0 I agree with everything you’re proposing. We can review this after 57 release and after we’ve validated that the email first flow is beneficial on firstrun. If we change too many things, it’ll prevent us from efficiently measuring the impact of email first. As a reminder, the hypothesis of his test is: >If we ask users for their email address only on /firstrun and redirect them to the appropriate page (/signin or /signup), we will see a greater percentage of users complete both flows because the flow will be simpler for all users.
(In reply to Alex Davis [:adavis] [PM FxA+Sync] from comment #3) > If we change too many things, it’ll prevent us from efficiently measuring > the impact of email first. Could we A/B/C test Ryan's proposal against the other two possibilities? Firstrun is by far the highest volume integration we have, getting sufficient numbers for more than one bucket shouldn't be a problem. I agree with Ryan that the iframe has been a bane for us, and all the content server work is already done to enable what Ryan proposes, the only portion that remains is the firstrun side.
If the Moz org team is able to do A/B/C, I would not be opposed to the last variation in rfeeley's video which asks for the email and passes it to FxA so long as we use the EXACT SAME COPY in version B and C to isolate the impact of removing the iFrame without any other variables. This would could help us better understand the impact of asking for email first, just in a different way. What is missing from Ryan's proposal is: - what will it look like after the user enters his email? - how will that experience lead users into the aha moment with the animation that leads to Activity Stream with the new onboarding tour? Rfeeley and I worked on a feature document linked above (i.e. "test recipe" for the marketers), if we are to propose a new variation please add details of the flow there. If we can't easily answer the questions and outline the full flow this week, I will ask that we post-pone the extra variation. Fx 57 is released in 4-5 weeks, the onboarding team has been doing user testing before the release to make sure there is nothing obviously broken with onboarding, we don't have much wiggle room here.
(In reply to Alex Davis [:adavis] [PM FxA+Sync] from comment #5) > > If we can't easily answer the questions and outline the full flow this week, > I will ask that we post-pone the extra variation. Fx 57 is released in 4-5 > weeks, the onboarding team has been doing user testing before the release to > make sure there is nothing obviously broken with onboarding, we don't have > much wiggle room here. Understood.
Per the FxA meeting this morning, we're going to keep it simple with 57 around the corner. Let's just test what was outlined in the bug description and Comment 1. See RFeeley's attached screenshot. To make things clear, here is the recap: **SUMMARY** COPY Since the current copy says to login and that becomes irrelevant when we just ask for email, we propose a very simple copy change for the test to: >Take Firefox with You >Get your bookmarks, history, passwords, and other settings on all your devices. TRACKING To track the full FxA funnel, please past different UTM params to our iframe: -For the control, I propose utm_source=firstrun-login -For the treatment, I propose utm_source=firstrun-emailfirst SCREENSHOT https://bugzilla.mozilla.org/attachment.cgi?id=8915343 iFRAME In order to load "email first" instead of current iframe you would need to load /?action=email rather than /signin
Additional details that I forgot and added to test document: Locale: EN-* Audience: 10% of traffic Variations: 50/50 (5% each)
(In reply to Alex Davis [:adavis] [PM FxA+Sync] from comment #7) > Per the FxA meeting this morning, we're going to keep it simple with 57 > around the corner. > > Let's just test what was outlined in the bug description and Comment 1. See > RFeeley's attached screenshot. > > To make things clear, here is the recap: > > **SUMMARY** > > COPY > Since the current copy says to login and that becomes irrelevant when we > just ask for email, we propose a very simple copy change for the test to: > > >Take Firefox with You > >Get your bookmarks, history, passwords, and other settings on all your devices. > > TRACKING > To track the full FxA funnel, please past different UTM params to our iframe: > -For the control, I propose utm_source=firstrun-login > -For the treatment, I propose utm_source=firstrun-emailfirst > > SCREENSHOT > https://bugzilla.mozilla.org/attachment.cgi?id=8915343 > > iFRAME > In order to load "email first" instead of current iframe you would need to > load /?action=email rather than /signin Will the logo on this be updated for 57?
Flags: needinfo?(adavis)
> Will the logo on this be updated for 57? The logo displayed in the FxA signin form? Yes, once [1] lands we should detect when the user is on 57+ and show the new logo. [1] https://github.com/mozilla/fxa-content-server/pull/5603/files
Clearing ni? since :rfkelly answered
Flags: needinfo?(adavis)
Assignee: nobody → agibson
Status: NEW → ASSIGNED
:agibson - I should have assigned this to myself, I've already got a WIP! Lemme push it up to get feedback and start to ask questions.
:agibson - https://github.com/mozilla/bedrock/pull/5316 is the WIP PR I was talking about.
Ah ok, Eric asked me to pick this up - but if you've already got a WIP by all means :)
Assignee: agibson → stomlinson
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Commits pushed to master at https://github.com/mozmeao/www-config https://github.com/mozmeao/www-config/commit/42e874d1635765da37cfe4326979d7fbaf479f46 [bug 1405822] Enable /firstrun email experiment https://github.com/mozmeao/www-config/commit/a13fc1e0506bc614dfaa2f48c0ae85ba302fb42a Merge pull request #63 from alexgibson/enable-firstrun-email-experiment [bug 1405822] Enable /firstrun email experiment
Just checking in here - how are we doing in terms of data? Is there an expected date when this experiment can be shut off?
Flags: needinfo?(adavis)
FYI :adavis is out this week, returning on the 15th. I tried to have a look at the metrics so far, but unfortunately there seems to be a problem with our redshift cluster and I haven't yet been able to execute the necessary queries. I expect we should have more than enough data though, so if there's a pressing need to turn off the experiment please feel free to do so. Otherwise we can double-check with Alex when he returns.
(In reply to Ryan Kelly [:rfkelly] from comment #19) > FYI :adavis is out this week, returning on the 15th. > > I tried to have a look at the metrics so far, but unfortunately there seems > to be a problem with our redshift cluster and I haven't yet been able to > execute the necessary queries. I expect we should have more than enough > data though, so if there's a pressing need to turn off the experiment please > feel free to do so. Otherwise we can double-check with Alex when he returns. We do have some other experiments coming on the horizon, but I think we're fine to wait until the 15th. Thanks!
I've started gathering the results of this experiment here: https://sql.telemetry.mozilla.org/dashboard/fxa-experiment-firstrun-emailfirst
Small update: Leif and I are still doing the full analysis. We were digging in more in Amplitude for a couple of hours yesterday but there were some metric anomalies (which :pbooth is digging into). Leif will fork :rfkelly 's query and change the cohorting to use the experiment cohorts rather than the utm params being used in it since there may be problems with the utm params too according to :pbooth Leaving ni? on me
:agibson , it is our recommendation to roll-out the treatment (email-first version). We've logged our analysis here and are making sure to get it to everyone who is concerned: https://docs.google.com/document/d/1seUQEXOR-JGD6ff77iAP8lzDjW-BPengXugUzo-p2UQ/edit# In summary, more users were syncing on the new version and overall more users were retained in the long run. There were just some tradeoffs as always between logins and registrations.
Flags: needinfo?(adavis)
(In reply to Alex Davis [:adavis] [PM FxA+Sync] from comment #23) > :agibson , it is our recommendation to roll-out the treatment (email-first > version). > > We've logged our analysis here and are making sure to get it to everyone who > is concerned: > https://docs.google.com/document/d/1seUQEXOR-JGD6ff77iAP8lzDjW-BPengXugUzo- > p2UQ/edit# > > In summary, more users were syncing on the new version and overall more > users were retained in the long run. There were just some tradeoffs as > always between logins and registrations. I'll flag this with Eric and make sure it gets on our board. Thanks
Flags: needinfo?(erenaud)
Alex - I confirmed with the Retention team that we're good to go with taking ?v=b live to 100% of audience, as soon as possible.
Flags: needinfo?(erenaud)
Assignee: stomlinson → agibson
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → ASSIGNED
This is now up on demo3: https://www-demo3.allizom.org/en-US/firefox/58.0/firstrun/ Note: to test on demo you'll need to both enable UITour [1] and set up a profile to test the iframe on demo [2]. When testing a signup for a new account, I'm not receiving any verification emails. I guess this is likely due to the staging server config, but I'd like to verify that the full flow works as expected before pushing this forward to production. Can someone please investigate? (I remember when we tested the experiment, Shane had this running against his own instance and not the regular staging server). [1] http://bedrock.readthedocs.io/en/latest/uitour.html#local-development [2] http://bedrock.readthedocs.io/en/latest/firefox-accounts.html#demo-server-testing
Flags: needinfo?(adavis)
Flags: needinfo?(rfkelly)
> I'd like to verify that the full flow works as expected before pushing this forward to production I just tried it out and this worked end-to-end for me, using the following steps: * Create a new profile * Add pref "browser.uitour.testingOrigins" with value "https://www-demo3.allizom.org" * Add pref "identity.fxaccounts.autoconfig.uri" with value "https://accounts.stage.mozaws.net" * Go to about:preferences#sync and click "sign in" * Check that it correctly loaded the signin page from https://accounts.stage.mozaws.net, but don't actually sign in * Restart the browser * Visit https://www-demo3.allizom.org/en-US/firefox/58.0/firstrun/ * Go through signup process and get connected to sync So AFAICT this is looking :thumbsup:!
Flags: needinfo?(rfkelly)
(In reply to Ryan Kelly [:rfkelly] from comment #28) > > I'd like to verify that the full flow works as expected before pushing this forward to production > > I just tried it out and this worked end-to-end for me, using the following > steps: > > * Create a new profile > * Add pref "browser.uitour.testingOrigins" with value > "https://www-demo3.allizom.org" > * Add pref "identity.fxaccounts.autoconfig.uri" with value > "https://accounts.stage.mozaws.net" > * Go to about:preferences#sync and click "sign in" > * Check that it correctly loaded the signin page from > https://accounts.stage.mozaws.net, but don't actually sign in > * Restart the browser > * Visit https://www-demo3.allizom.org/en-US/firefox/58.0/firstrun/ > * Go through signup process and get connected to sync > > So AFAICT this is looking :thumbsup:! Ok, I verified this works using a different email address via the steps above. Thanks for testing!
Flags: needinfo?(adavis)
Commit pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/3cd528bd9f17592c7a71aa9a2a5362840e5cb84e [fix bug 1405822] Set Firefox Accounts email-first form as default on /firstrun
Status: ASSIGNED → RESOLVED
Closed: 7 years ago → 7 years ago
Resolution: --- → FIXED
Verified in production https://www.mozilla.org/en-US/firefox/58.0/firstrun/
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: