Closed Bug 1195573 Opened 9 years ago Closed 7 years ago

The first run page inviting to create a firefox account should make it clearer that it's not required

Categories

(www.mozilla.org :: Pages & Content, defect)

Product:
www.mozilla.org
Component:
Pages & Content
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
Future

People

(Reporter: glandium, Unassigned)

References

Details

(Whiteboard: [fxgrowth])

Attachments

(1 file)

No thank you screenshot
135.91 KB, image/png
Details 
The pages says, after installing Firefox:

"Almost done!

Sign in to Firefox and you're good to go."

Nowhere on the page is there anything saying this is actually optional.

At the very least, there should be a "No, thank you" kind of link under "Sign up" or under "Already have an account".
Component: Tours → Pages & Content
Product: Firefox → www.mozilla.org
Target Milestone: --- → Future
Version: 38 Branch → Production
I've seen a few individual reports on this now - CC'ing :cmore and :habber for thought.

I understand that we want to encourage new users to sign up for Firefox Accounts, but I'm curious if we have any data on the effect this might have re: potential abandonment from new users who think a signup is required? My gut feeling is that this type of abandonment can be pretty common in the mobile app world, but on desktop it could be entirely different. It would be interesting to know if we have any sort of data on this?

Either way, altering the language slightly might still be worth considering.
I have an open bug 1198373 with the metrics team to understand what impact Firefox accounts has on MAUs and retention to help answer this question. I can't answer it purely on engagement with the first run page. Specifically comparing desktop profiles with no accounts, desktop profiles with an account and one device, and desktop profiles with syncing multiple devices.
Whiteboard: [fxgrowth]
(In reply to Chris More [:cmore] from comment #2)
> I have an open bug 1198373 with the metrics team to understand what impact
> Firefox accounts has on MAUs and retention to help answer this question. I
> can't answer it purely on engagement with the first run page. Specifically
> comparing desktop profiles with no accounts, desktop profiles with an
> account and one device, and desktop profiles with syncing multiple devices.

Cmore - 
I would also be interested testing the impact of the language used in the invitation to create an Account on MAUs.  Is it possible that changing copy to better help the user understand that creating an account is not required would not have an impact on MAUs?
Thx,
Jen
The language used on this page is partly an artifact of it being a web page that opens after Firefox is fully installed. If we implement the design in https://bugzilla.mozilla.org/show_bug.cgi?id=670989#c5 we can easily experiment with different language with possibly less risk of people ignoring the page (like they tended to do when we used a "Welcome to Firefox" headline).
What if we did a a/b test with a "Not now" link at the bottom that redirects them to about:home? It is possible that the "Not now" would be as suffice as a "skip" and get them on their way to doing other tasks.
(In reply to Chris More [:cmore] from comment #5)
> What if we did a a/b test with a "Not now" link at the bottom that redirects
> them to about:home? It is possible that the "Not now" would be as suffice as
> a "skip" and get them on their way to doing other tasks.

Sounds good to me. We could also try having that link take people to about:newtab.
(In reply to Verdi [:verdi] from comment #6)
> (In reply to Chris More [:cmore] from comment #5)
> > What if we did a a/b test with a "Not now" link at the bottom that redirects
> > them to about:home? It is possible that the "Not now" would be as suffice as
> > a "skip" and get them on their way to doing other tasks.
> 
> Sounds good to me. We could also try having that link take people to
> about:newtab.

Note: we currently have no API to open either about:home or about:newtab
Oh interestingly, about:home as a regular link seems to work. Testing about:newtab does not. 

Most internal pages like this are not accessible as regular links. Perhaps about:home is an exception?
(In reply to Alex Gibson [:agibson] from comment #8)
> Oh interestingly, about:home as a regular link seems to work. Testing
> about:newtab does not. 
> 
> Most internal pages like this are not accessible as regular links. Perhaps
> about:home is an exception?

Yup. We've tried it before and it works to link to about:home via an anchor or even a window.location JS redirect.
We could probably do an A/B test on /firstrun/ to inject a link on the page to see how many people would click it. 

It would be fairly straightforward to add a few different "Not now" links from a simple text link to a button that links to about:home.
:openjck is going to set up a test to see how this link would be used on the page.
Great idea for an experiment. I can see how non-technical users might be confused by the current language and drop off.

We could alternatively use Google Analytics to measure clicks to the link, but that assumes that we know we want it. What statistics would help us determine if we want to launch the link? Maybe account creation rate, or MAUs, or both?
(In reply to John Karahalis [:openjck] from comment #12)
> Great idea for an experiment. I can see how non-technical users might be
> confused by the current language and drop off.
> 
> We could alternatively use Google Analytics to measure clicks to the link,
> but that assumes that we know we want it. What statistics would help us
> determine if we want to launch the link? Maybe account creation rate, or
> MAUs, or both?

We could do one with a link, one without a link, and see if the has any impact on sign up rates. Sign up rates may be tricky to measure with Optimizely since it is in an iframe, but it does send a JS event back out of the iframe on sign up, which we should be able to catch. We'll see :)

Variations:

A: no link (control)
B: "Not now" link that goes to about:home

Goal: Sign up rates

Audience: En-US users
Thanks for that clarification.

Mana page:
https://bugzilla.mozilla.org/show_bug.cgi?id=1195573

Optimizely experiment blocked on bug 1209687.
No longer blocks: 1209687
Depends on: 1209687
Code for review:
https://github.com/mozilla/mozorg-js-external/pull/2/files

After the code is reviewed, Optimizely will be enabled on the firstrun page and the experiment will be created.
(In reply to John Karahalis [:openjck] from comment #16)
> Code for review:
> https://github.com/mozilla/mozorg-js-external/pull/2/files
> 
> After the code is reviewed, Optimizely will be enabled on the firstrun page
> and the experiment will be created.

Let's not get this code reviewed For now as the process is not ready for it. Peer review comes before code reviews.

Steps:

1) Create mana page
2) Create Optimizely experiment (you don't need optimizely enabled on a page to create a test)
3) Add the Optimizely code to the mana page
4) Add the test to the backlog spreadsheet and provide a link to the mana
5) Needinfo cmore to review the experiment to ensure it is doing the right things and is setup correctly.
6) After we settle on the changes that we need, update the PR from comment 16, and get it code reviewed from a bedrock dev.
7) After code is reviewed and approved, bedrock devs enable optimizely on the page.
8) Start test
Chris, what's the status of the Optimizely test you we're looking to run here? Do we have any actions we can take to resolve this bug?
(In reply to Alex Gibson [:agibson] from comment #19)
> Chris, what's the status of the Optimizely test you we're looking to run
> here? Do we have any actions we can take to resolve this bug?

We didn't get around to running this test, but I will it back on the list of items to test soon. Thanks for the ping.
Attached image No thank you screenshot — 
We tested adding a "No thank you!" link below the form that linked to about:home and 5.8% of users clicked it.
(In reply to Chris More [:cmore] from comment #21)
> Created attachment 8756433 [details]
> No thank you screenshot
> 
> We tested adding a "No thank you!" link below the form that linked to
> about:home and 5.8% of users clicked it.

So could we add it to the page to help the situation, seeing as it indicates something to at least a significant (but minor) number of users?
See Also: → 1274627
(In reply to Alex Gibson [:agibson] from comment #22)
> (In reply to Chris More [:cmore] from comment #21)
> > Created attachment 8756433 [details]
> > No thank you screenshot
> > 
> > We tested adding a "No thank you!" link below the form that linked to
> > about:home and 5.8% of users clicked it.
> 
> So could we add it to the page to help the situation, seeing as it indicates
> something to at least a significant (but minor) number of users?

Here's the results of the English global a/b test:

* Saying "No Thanks" had the highest link engagement rate of 4.8% and has highest negative impact FxA submissions rates by -9.4%

* Saying "Ask me later" has lowest link engagement ratee of 3.9% and the lowest negative impact to FxA submissions by -7.8%.

* Anything said below the form lowers the FxA submission rate by 7-10%.

* Given that email, snippet, and in-product mentions creating a Firefox account later, "Ask me later" seems to be the best option since it is likely they will see a future prompt about creating an account tied to a value proposition.

* All of the links below the form simply linked to about:home
See Also: → 1177248
What's the current plan for fixing this page? It's embarrassing that we're still misleading users like this.

FWIW, I don't think adding a small "No thanks" / "Ask me later" link is good enough. We need to replace "Almost Done!" with something more honest.
Flags: needinfo?(chrismore.bugzilla)
Flags: needinfo?(agibson)
(In reply to Dão Gottwald [:dao] from comment #24)
> What's the current plan for fixing this page? It's embarrassing that we're
> still misleading users like this.
> 
> FWIW, I don't think adding a small "No thanks" / "Ask me later" link is good
> enough. We need to replace "Almost Done!" with something more honest.

I'd like to fix this too. Chris, as we OK to add this "Ask me later" link below the form given the results in Comment 23? Until the time comes when this page is updated / redesigned given the main call to action, I think adding the link is a good interim step.
Flags: needinfo?(agibson)
(In reply to Dão Gottwald [:dao] from comment #24)
> What's the current plan for fixing this page? It's embarrassing that we're
> still misleading users like this.
> 

I agree that this page has to be changed. In the next onboarding experiment - Bug 1322718 - we're going to replace this page primarily with a sign-in flow for users with an account. We're also going to test some other designs from the FxA team.
(In reply to Alex Gibson [:agibson] from comment #25)
> (In reply to Dão Gottwald [:dao] from comment #24)
> > What's the current plan for fixing this page? It's embarrassing that we're
> > still misleading users like this.
> > 
> > FWIW, I don't think adding a small "No thanks" / "Ask me later" link is good
> > enough. We need to replace "Almost Done!" with something more honest.
> 
> I'd like to fix this too. Chris, as we OK to add this "Ask me later" link
> below the form given the results in Comment 23? Until the time comes when
> this page is updated / redesigned given the main call to action, I think
> adding the link is a good interim step.

Yeah, "Ask me later" sounds good to me and feels very appropriate. Our testing of different versions of that, it will probably be clicked around 4% of the time, but it is still the right thing to do for various reasons.

As Verdi mentioned, there are lots of versions being tested now to create something better.
Flags: needinfo?(chrismore.bugzilla)
Depends on: 1347187
We're changing the headline of /firstrun from "Almost done!" to "Free as a fox" in Bug 1347187, so we can soon finally resolve this bug :)
Fixed in Bug 1347187
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: