Closed Bug 1150231 Opened 4 years ago Closed 4 years ago

Mozilla.org /firefox/firstrun/ - Development for 40.0

Categories

(www.mozilla.org :: Pages & Content, defect)

Production
defect
Not set

Tracking

(firefox40+ fixed)

RESOLVED FIXED
Tracking Status
firefox40 + fixed

People

(Reporter: cmore, Assigned: jpetto)

References

Details

(Whiteboard: [kb=1772975][fxgrowth])

Attachments

(2 files)

Placeholder bug for dev work for the improved sync first run flow.
Blocks: 1150222
Depends on: 1142046
jpetto: I was talking to Shane about two parameters on the fxa embedded form. For the iframe URL, we have the ability to populate two query parameters that is then pulled into the metrics.

1) campaign= [any string]

2) entrypoint = [any string on how they got to the form]

For example:

<iframe src="https://accounts.firefox.com/signup?campaign=fxa-embedded-form&entrypoint=firstrun"></iframe>

The URL I just made up, but the query parameters are correct.

Can we make this change to your iframe URL that you are working?
Flags: needinfo?(jon)
Yep, got it in now.
Flags: needinfo?(jon)
Summary: Mozilla.org /firefox/firstrun/ - Development for 39.0 → Mozilla.org /firefox/firstrun/ - Development for 40.0
Assignee: nobody → jon
OS: Mac OS X → All
Hardware: x86 → All
Whiteboard: [kb=1772975]
We need to get a demo of the embedded FxA form on first run on a demo server asap. The spring campaign sucked up a lot of time, but we need to get moving now. The Firefox product changes will also land in Firefox 40, so any testing will need to be done with Fx v 40+ browsers.
Flags: needinfo?(jon)
The Fx40 firstrun branch is up on demo4:

https://www-demo4.allizom.org/en-US/firefox/40.0/firstrun/

One note before testing: The FxA side is actively under development, meaning there may be occassional hiccups/inconsistencies on that end. If you should see an error or odd message and are able to replicate, please take a screenshot and comment on this bug.

On to testing...

You can view the sign-up/in forms without any browser/profile config changes, but to actually test signing up/in, you’ll need to make some modifications to your profile. In the interest of safety, my recommendation is to create a new profile for FxA testing:

1. Quit Firefox

2. Create a new profile for testing. I recommend naming it “FxA Test Demo":

    https://support.mozilla.org/kb/profile-manager-create-and-remove-firefox-profiles

3. In the new profile folder, create a new file named "user.js", paste in the
contents of the following gist, and save:

    https://gist.github.com/jpetto/c850ed3f63328b85b9f1

4. Start Firefox using the new profile you created in step 2

5. Verify FxA settings have been updated:

    i: Navigate to about:config
    ii: Enter "identity.fxaccounts" into the search form - you should see the
        values specified in user.js listed. Here is a screenshot:

        https://cloudup.com/cNOziFzhTC4

6. Navigate to the first run page and sign up.
Flags: needinfo?(jon)
:cmore - Are we going to be using funnelcake params on this firstrun test as well? If so, can you list the numbers and variations?

The one copy variation we have remaining from 38.0.5 is the subhead. From the 38.0.5 firstrun:

f=37: Set up Sync and you’re good to go.
f!=37: Sign in to Firefox and you’re good to go.

Video variations will still be random, unless the v=1|2 param is set. v=1 displays the video overlay, v=2 displays the video thumbnail under the main content.
Flags: needinfo?(chrismore.bugzilla)
(In reply to Jon Petto [:jpetto] from comment #5)
> :cmore - Are we going to be using funnelcake params on this firstrun test as
> well? If so, can you list the numbers and variations?
> 
> The one copy variation we have remaining from 38.0.5 is the subhead. From
> the 38.0.5 firstrun:
> 
> f=37: Set up Sync and you’re good to go.
> f!=37: Sign in to Firefox and you’re good to go.
> 
> Video variations will still be random, unless the v=1|2 param is set. v=1
> displays the video overlay, v=2 displays the video thumbnail under the main
> content.

No funnelcake with this one.

I would just keep all of the headlines as "Sign in to Firefox and you’re good to go." and just add the 50:50 split of the video overlay and below. This is basically what we have now on 38.0.5 without a funnelcake (since the funnelcake was shut off right at the start).

Yes, keep v=1 as the overlay and v=2 as the video below. We'll need to send the same GA variables too as 38.0.5 except we don't need to worry about the f= variations.

Does that work?
Flags: needinfo?(chrismore.bugzilla) → needinfo?(jon)
:bsilverberg: can you help with QA'ing this embedded form? How to test is listed in comment 4. We don't need to test the video overlay, but we need to specifically test the embedded form and anything inside the white DIV that contains the Firefox Account form and options. Thanks in advance!
Flags: needinfo?(bob.silverberg)
:rbillings: you and bob can fight it out on who wants to help us with QA here. :)
Flags: needinfo?(rbillings)
I'll take it :)
Flags: needinfo?(rbillings)
(In reply to Chris More [:cmore] from comment #6)
> 
> No funnelcake with this one.
> 
> I would just keep all of the headlines as "Sign in to Firefox and you’re
> good to go." and just add the 50:50 split of the video overlay and below.
> This is basically what we have now on 38.0.5 without a funnelcake (since the
> funnelcake was shut off right at the start).
> 
> Yes, keep v=1 as the overlay and v=2 as the video below. We'll need to send
> the same GA variables too as 38.0.5 except we don't need to worry about the
> f= variations.
> 
> Does that work?

Yep, all good. Updates have been made and pushed to demo4:

- Removed custom headlines in favor of in-frame headlines
- Removed funnelcake logic
- Made "Sign in to Firefox and you’re good to go." the one and only left-side sub-headline.

https://www-demo4.allizom.org/en-US/firefox/40.0/firstrun/?v=2
Flags: needinfo?(jon)
Oops, sorry cmore. I was off yesterday and it looks like rbillings grabbed this one. Feel free to ask again in the future.
Flags: needinfo?(bob.silverberg)
:jpetto: check iut v=1 and video below issue I am seeing: 

https://www-demo4.allizom.org/en-US/firefox/40.0/firstrun/?v=1

http://cl.ly/image/2t1z0H1H0K0D

v=2 shows the video below fine, but v=1 has an overlay issue.
Flags: needinfo?(jon)
(In reply to Chris More [:cmore] from comment #12)
> :jpetto: check iut v=1 and video below issue I am seeing: 
> 
> https://www-demo4.allizom.org/en-US/firefox/40.0/firstrun/?v=1
> 
> http://cl.ly/image/2t1z0H1H0K0D
> 
> v=2 shows the video below fine, but v=1 has an overlay issue.

Ah, the modal CSS file got refactored/renamed last week. I had a cache of the old version locally, so never saw the issue. Fixed now on demo4.
Flags: needinfo?(jon)
:jpetto: what do you think about margin-top: 10px instead of margin-top: 20px for the rightcol? The alignment with the 10px with the headline looks slightly better than 20px, but I wanted to get your opinion.
Flags: needinfo?(jon)
Yep, agree that 10px looks better. Updated.
Flags: needinfo?(jon)
Verified signing in, editing, deleting accounts, creating new accounts
Shane: We want to be able to capture people successfully submitting the Firefox accounts form from the /firstrun/ page as a Google Analytics event on the page. This would mean that the FxA iframed page would need to push triggers back to the parent frame so that Google Analytics can set them as a local GA event. What are all the events that are being sent from the FxA form back to the parent iframe so that :jpetto can code them up to be sent to Google Analytics?
Flags: needinfo?(stomlinson)
The only interaction-related event I'm seeing is "verification_complete" after the user's email address has been confirmed. I believe this is the only user interaction event intended at this point.

Asked in Shane's PR (https://github.com/mozilla/fxa-content-server/pull/2705#issuecomment-122402549) if we can listen to the sign up form as well, both for successful and unsuccessful submissions.
:cmore - What do you think about the requested notifications above? An unsuccessful submission would be clicking submit with invalid form data (email is incorrect/taken, password too short, age invalid).
Flags: needinfo?(chrismore.bugzilla)
I would like an event for both form submit and verification, because it is possible that someone will verify after they submit, close the tab, and then we'll never capture any conversion on the page.
Flags: needinfo?(chrismore.bugzilla)
(In reply to Jon Petto [:jpetto] from comment #19)
> :cmore - What do you think about the requested notifications above? An
> unsuccessful submission would be clicking submit with invalid form data
> (email is incorrect/taken, password too short, age invalid).

I think the invalid form data would be great for future optimizations and to have that data would great, but the successful submit is even more important.
We now have the following events coming from the iframe:

- loaded: iframe has successfully loaded on the page.

- signup_must_verify: User has signed up successfully, but has yet to verify their email address.

- verification_complete: User has verified their email address.

- login: User with an existing FxA account has logged in.

:cmore or :garethc - Can you provide GA details for the relevant events above?
Flags: needinfo?(stomlinson)
The messages sent to the firstrun page are documented at [1] and will be kept up to date as new events are added.

[1] - https://github.com/mozilla/fxa-content-server/blob/master/docs/relier-communication-protocols/firstrun.md
As Jon notes in #c18, a request was made to report unsuccessful signup attempts. This required a larger change than I could add to train 42, and should be part of train 43. Progress can be tracked at https://github.com/mozilla/fxa-content-server/issues/2791
garethc: please fill out this spreadsheet with all of the meta data that should be populated in the data layer pushes so we can see key UX events to help us determined if the new experience is working: https://docs.google.com/spreadsheets/d/1_TwztagtHliQsL4LgNmXrS0pI1NX2SCU2Uokig4fA8E/edit#gid=0

Then give jpetto the exact code he should trigger in the page. We should make the naming of these events more universal since this page will stay like this for a while and it isn't a fx40 specific version.
Flags: needinfo?(garethcull.bugs)
:cmore and :petto, can we define what constitutes an unsuccessful signup?

A lot of users could hit some sort of form validation error and still successfully sign up. Reporting validation errors to the parent for users who still successfully complete the signup seems like overkill. It seems like the important event is whether the user actually abandons the signup process.
Flags: needinfo?(chrismore.bugzilla)
(In reply to Shane Tomlinson [:stomlinson] from comment #26)
> :cmore and :petto, can we define what constitutes an unsuccessful signup?
> 
> It seems like the important event is whether the user actually abandons the signup
> process.

Agreed. Could we track this without knowing if a user tried to submit the form?
Updated testing instructions for demo4:

1. Quit Firefox

2. Create a new profile for testing. I recommend naming it "Testing FxA Stage":

    https://support.mozilla.org/kb/profile-manager-create-and-remove-firefox-profiles

3. In the new profile folder, create a new file named "user.js", paste in the
contents of the following gist, and save:

    https://gist.github.com/jpetto/f46f1850737b7a838cad

4. Start Firefox using the new profile you created in step 2

5. Verify FxA settings have been updated:

    i: Navigate to about:config
    ii: Enter "identity.fxaccounts" into the search form - you should see the
        values specified in user.js listed.

6. Navigate to the first run page and sign up.
The URL to demo4 first run is:

https://www-demo4.allizom.org/en-US/firefox/40.0/firstrun/
Flags: needinfo?(chrismore.bugzilla)
(In reply to Shane Tomlinson [:stomlinson] from comment #26)
> :cmore and :petto, can we define what constitutes an unsuccessful signup?
> 
> A lot of users could hit some sort of form validation error and still
> successfully sign up. Reporting validation errors to the parent for users
> who still successfully complete the signup seems like overkill. It seems
> like the important event is whether the user actually abandons the signup
> process.

:stomlinson:

I still like validation errors, because it is then something we can work together on to improve. Alex Davis and I have hypotheses on the regular FxA form to help reduce friction when signing. If a lot of people are getting validation errors, we can see if those validation errors are impacting conversions or if we make changes in the future, we can see if that helps/hurts successful submissions.

Do we the validation error data already in FxA so we can understand what percentage of people get validation errors?
cmore,

firstrun Video and FxA Events are now configured in GTM and are coming through in GA:
http://cl.ly/image/2Z2M1K0d1D24
Flags: needinfo?(garethcull.bugs)
(In reply to Gareth Cull [:garethc] from comment #31)
> cmore,
> 
> firstrun Video and FxA Events are now configured in GTM and are coming
> through in GA:
> http://cl.ly/image/2Z2M1K0d1D24

Sounds good! thanks
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Chris, I guess we are good to go for next week? Do you need help from QE to verify that?
Flags: needinfo?(chrismore.bugzilla)
(In reply to Sylvestre Ledru [:sylvestre] from comment #33)
> Chris, I guess we are good to go for next week? Do you need help from QE to
> verify that?

I think we are good, but if you could also do any extra verification, it would be much appreciated.
Flags: needinfo?(chrismore.bugzilla) → needinfo?(sledru)
Florin, do you know who could help with the First run page test? Thanks
Flags: needinfo?(sledru) → needinfo?(florin.mezei)
Catalin is already working on this and will soon post the test results here.
Flags: needinfo?(florin.mezei)
Attached file logs.zip
I've tested this using:

FF 40
Build Id: 20150803103853
OS: Win 7 x64

I've followed the instructions from comment 28, the the preferences were successfully modified, the singup form for Firefox Account appeared on the first run page but I encountered several issues:

1. The verification of the email was not successful on the first try, I had to restart FF several times and verify the email again until it was finally successful.
2. After I managed to verify the email the sync was not working.

I've attached the sync-logs.
I tested this on Win10/FF40 and MacOSX/FF40 this morning - I was unable to create an account and unable to sign in with an existing account on both browsers. I got a spinning wheel with "Working" that never resolved. I did double check the account set up process & config settings.
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/331098e77b55fd3c68d6474e527f3c2e28dd46b4
[fix bug 1150231] Fx40 firstrun experience.

https://github.com/mozilla/bedrock/commit/dbb88f928354050bedcf33955d26e0ab4411532d
Merge pull request #3168 from mozilla/bug-1150231-firstrun-fx40

[fix bug 1150231] Fx40 firstrun experience.
:jpetto - the user.js you pointed to has settings for https://accounts.stage.mozaws.net/, the demo server cmore pointed to is https://www-demo4.allizom.org/. That would cause the behavior :rbillings is setting (web channel origin mismatch, so the browser does not respond to messages).
Flags: needinfo?(jon)
Ignore me, I'm trying to dig in here to figure out what possible problems are. I'll keep my trap shut before I can reproduce or come up with better theories.
Flags: needinfo?(jon)
Just created a fresh profile using the instructions in comment 28 and tested sign up and sign in successfully on Fx40 (OS X Yosemite).
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
:rbillings - the behavior you describe (the Waiting message) is often because |identity.fxaccounts.remote.webchannel.uri| is incorrect in about:config. Can you confirm your profile's user.js settings match those of the user.js jpetto pointed to at [1]? I just ran through both the signup and signin flows without a hitch by:

1. Create a new Firefox profile.
2. Download the raw version of that user.js from [2]. Move user.js to the profile directory created in (1).
3. Open the profile created in (1) in Firefox 40 (current Beta). Firefox 40 is important, Firefox 39 will *not* work.
4. Sign up to FxA.
5. Open the verification link, go back to the firstrun page. After a few seconds, the screen should transition to a message that says signup was successful.

[1] - https://gist.github.com/jpetto/f46f1850737b7a838cad
[2] - https://gist.githubusercontent.com/jpetto/f46f1850737b7a838cad/raw/d77eab0faf6d2d2d3a4bc9fa50b7b18dd1482a19/user.js
This is working - my issue was with the profile manager, which makes it a non-issue. Best outcome!
Woo hoo! Thanks everyone for the additional testing.
Attached image not_verified.jpg
I've retested using FF 40 build id:20150803103853 and Win 7 x64 and I still didn't manage to verify the email.
I've used the steps from comment 28 and the following link https://www-demo4.allizom.org/en-US/firefox/40.0/firstrun/

After I verify the email the form from firstrun page is displaying me as logged in to the account but the about:preferences#sync says that I still have to verify the email( I've attached a screenshot for a better understanding of the state).In this state I can't start a sync.
agibson and I just tested on prod (mozilla.org/accounts.firefox.com) and verified that sign up, sign out, and sign in are all working as expected. The waffle switch was turned off after our tests. We should be all set for launch tomorrow.
Jon, good to know it is working on your side.
However, any idea why Catalin keep having this issue? This is a bit scary for tomorrow release and I am not sure what we can go live with this page.
Flags: needinfo?(jon)
(In reply to Sylvestre Ledru [:sylvestre] from comment #48)
> Jon, good to know it is working on your side.
> However, any idea why Catalin keep having this issue? This is a bit scary
> for tomorrow release and I am not sure what we can go live with this page.

My guess is that :VarCat's issues are due to the profile manipulation needed for testing on demo4.

I'd recommend another test, but this time on bedrock stage: https://www.allizom.org/firefox/40.0/firstrun/

Testing on stage doesn't require any profile manipulation (as the FxA embed points to FxA prod), though you still may want to create a fresh profile to better simulate real world interaction.

:VarCat - can you test against bedrock stage?
Flags: needinfo?(jon) → needinfo?(catalin.varga)
Is this expected to be activated using the whatsnew mechanism? Or something else?
Flags: needinfo?(chrismore.bugzilla)
(In reply to Ben Hearsum (:bhearsum) from comment #50)
> Is this expected to be activated using the whatsnew mechanism? Or something
> else?

This iteration of firstrun is unique in that we have a built-in fallback on mozilla.org should the FxA integration fall down for any reason.

When Fx40 is launched, we will turn on a waffle switch on mozilla.org that will enable the firstrun template with the embedded FxA iframe. If for any reason the FxA integration fails, we will turn off the waffle switch, which will result in Fx40 users receiving the 38.0.5 firstrun templates.
Flags: needinfo?(chrismore.bugzilla)
(In reply to Jon Petto [:jpetto] from comment #51)
> (In reply to Ben Hearsum (:bhearsum) from comment #50)
> > Is this expected to be activated using the whatsnew mechanism? Or something
> > else?
> 
> This iteration of firstrun is unique in that we have a built-in fallback on
> mozilla.org should the FxA integration fall down for any reason.
> 
> When Fx40 is launched, we will turn on a waffle switch on mozilla.org that
> will enable the firstrun template with the embedded FxA iframe. If for any
> reason the FxA integration fails, we will turn off the waffle switch, which
> will result in Fx40 users receiving the 38.0.5 firstrun templates.

Sounds like I got myself mixed up. This is /firstrun/, meaning that it's displayed to users launching Firefox for the first time (or from a new profile)?
(In reply to Ben Hearsum (:bhearsum) from comment #52)
> (In reply to Jon Petto [:jpetto] from comment #51)
> > (In reply to Ben Hearsum (:bhearsum) from comment #50)
> > > Is this expected to be activated using the whatsnew mechanism? Or something
> > > else?
> > 
> > This iteration of firstrun is unique in that we have a built-in fallback on
> > mozilla.org should the FxA integration fall down for any reason.
> > 
> > When Fx40 is launched, we will turn on a waffle switch on mozilla.org that
> > will enable the firstrun template with the embedded FxA iframe. If for any
> > reason the FxA integration fails, we will turn off the waffle switch, which
> > will result in Fx40 users receiving the 38.0.5 firstrun templates.
> 
> Sounds like I got myself mixed up. This is /firstrun/, meaning that it's
> displayed to users launching Firefox for the first time (or from a new
> profile)?

That is correct. :)
Thanks everyone on getting this live! We'll report on data later this week and next.
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Hi Jon, I've tested again using FF 40  with both https://www.allizom.org/firefox/40.0/firstrun/ and https://www.mozilla.org/en-US/firefox/40.0/firstrun/ and in both cases the accounts were successfully created and verified.
Flags: needinfo?(catalin.varga)
Thanks, that is making me happy for this release ;)
how's this for utm parameters?

utm_campaign=fxa-embedded-flow
utm_source=firstrun
utm_medium=referral
utm_campaign=fxa-embedded-form
utm_source=firstrun
utm_medium=referral
Opened a PR to update utm params: https://github.com/mozilla/bedrock/pull/3204
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/d858c433db5f4b978af275c1ff8dd9480a1e00b6
[fix bug 1150231] Update FxA iframe URL params.

https://github.com/mozilla/bedrock/commit/c4f69bbbd1ebd50cc5a145b470ce715b866dbada
Merge pull request #3204 from jpetto/bug-1150231-update-fxa-iframe-url-params

[fix bug 1150231] Update FxA iframe URL params.
utm_campaign=fxa-embedded-form
utm_source=firstrun
utm_medium=referral
utm_content=fx-version-[major version]

example:

utm_campaign=fxa-embedded-form
utm_source=firstrun
utm_medium=referral
utm_content=fx-version-40.0
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/064232cbc1006b750c80fdd73013a51f3206c5c3
[fix bug 1150231] Add utm_content param to iframe URL.

https://github.com/mozilla/bedrock/commit/2b4e13566e8069e2aeef707727f9c55a74e85349
Merge pull request #3205 from jpetto/bug-1150231-add-utm-content-param-fxa-iframe

[fix bug 1150231] Add utm_content param to iframe URL.
Whiteboard: [kb=1772975] → [kb=1772975][fxgrowth]
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/e3af2a4153cff06b0c874d698f613e685852312e
Remove fx40-firstrun waffle switch. Bug 1150231.

https://github.com/mozilla/bedrock/commit/97ba2ebb9476885a6b460a7ab3ad40728841ed54
Merge pull request #3467 from jpetto/bug-1150231-remove-fx40-firstrun-waffle

Remove fx40-firstrun waffle switch. Bug 1150231.
You need to log in before you can comment on or make changes to this bug.