1200277 - Final copy for First Run a/b testing

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: Screen Shot 2015-08-28 at 1.27.04 PM.png

We're moving forward with a couple options for testing improvements to our First run. I think we need some help with the strings/copy in order to fully communicate the message.

NI-ing Matej here for his usual awesome copy :)

Matej, we currently have a 1 slide experience for our First Time users. We would like to add 1 more slide that gives the user a chance to "import" their history and bookmarks. This feature is currently under Settings > Customize > "Import from Android - Import bookmarks and history from the native browser".


Thoughts? We are testing which order would be the most effective right now and attached are the mocks.

Feel free to ping if you have questions :D

Comment 1

[Matej Novak [:matej]]

In the version where the import screen comes first, the user isn't signed in to Sync yet. What happens when they click "Transfer to Firefox"?

In terms of copy, here's what I'm thinking:

Import your info
Transfer your bookmarks and browsing history from another computer or device.
Add it now


I swapped "transfer" and "import" because I felt that "Transfer your data" sounded too technical. The biggest challenge is the CTA. I like the idea of saying "Firefox" there, but then it could sound like you're importing data from a different browser. "Add it now" is a bit generic, but I like the way it plays off the headline.

Let me know what you think.

Comment 2

[Alex Davis [:adavis]]

(In reply to Matej Novak [:matej] from comment #1)
> In the version where the import screen comes first, the user isn't signed in
> to Sync yet. What happens when they click "Transfer to Firefox"?
> 
> In terms of copy, here's what I'm thinking:
> 
> Import your info
> Transfer your bookmarks and browsing history from another computer or device.
> Add it now
> 
> 
> I swapped "transfer" and "import" because I felt that "Transfer your data"
> sounded too technical. The biggest challenge is the CTA. I like the idea of
> saying "Firefox" there, but then it could sound like you're importing data
> from a different browser. "Add it now" is a bit generic, but I like the way
> it plays off the headline.
> 
> Let me know what you think.

The data is being transferred from the default Android browser on the same device so I don't feel like "... from another computer or device" applies here.

As for "Add it now", the button should complete this sentence in order to convert well, "I would like to...":
A) Import my settings
B) Import my history & passwords
C) Use my existing browser settings

We need to make sure it is not ambiguous.

Any chance you can take another pass at this?

Comment 3

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

(In reply to Matej Novak [:matej] from comment #1)
> In the version where the import screen comes first, the user isn't signed in
> to Sync yet. What happens when they click "Transfer to Firefox"?

It still works, you essentially inherit all that data from your previous browser (that you were using on your phone)

> In terms of copy, here's what I'm thinking:
> 
> Import your info
> Transfer your bookmarks and browsing history from another computer or device.
> Add it now
> 

Nice! although, AFAIK this only works on Android ATM, so it'd be "from the/your other/native browser" rather than computer or device.

> I swapped "transfer" and "import" because I felt that "Transfer your data"
> sounded too technical. The biggest challenge is the CTA. I like the idea of
> saying "Firefox" there, but then it could sound like you're importing data
> from a different browser. 

A mobile phone could have multiple browser apps so it is essentially doing that... maybe "data" in this case is still valid and also accurate?

> "Add it now" is a bit generic, but I like the way
> it plays off the headline.
> 
> Let me know what you think.

Yeah, I like the link between the title and the action item too :)

So, would the same copy apply to the reverse order as well?

Comment 4

[Matej Novak [:matej]]

Sorry, I misunderstood what was happening here. I thought it was the person importing their history and bookmarks from Firefox on another device, rather than from another browser on the same device. 

Given that Sync offers the same feature, could this be confusing to users? And what if the person gets a new Android phone and installs Firefox before even opening the native browser? It will be odd for them to see this message.

Given that we're aggressively trying to get people to sign up for an Account, wouldn't it be better to offer this up as another benefit of Sync (and therefore a reason to sign up) rather than importing their data from another browser?

Comment 5

[Alex Davis [:adavis]]

(In reply to Matej Novak [:matej] from comment #4)
> Sorry, I misunderstood what was happening here. I thought it was the person
> importing their history and bookmarks from Firefox on another device, rather
> than from another browser on the same device. 
> 
> Given that Sync offers the same feature, could this be confusing to users?
> And what if the person gets a new Android phone and installs Firefox before
> even opening the native browser? It will be odd for them to see this message.
> 
> Given that we're aggressively trying to get people to sign up for an
> Account, wouldn't it be better to offer this up as another benefit of Sync
> (and therefore a reason to sign up) rather than importing their data from
> another browser?

Importing settings from the default browser reduces friction for migration to FX. This is why we do it on desktop too.

In any case, we are A/B testing this feature and measuring user engagement with it. We don't want to assume it will bring value to the user.

I totally agree that there may be confusion around importing vs sync. This is where you can provide copy that will help clarify the difference.

I guess the best way to compare the 2:
-"Sync" allows to backup your existing settings or to pull them from another FX browser
-"Import" allows you to pull settings, history and bookmarks from another browser you are already using.

Comment 6

[Richard Newman [:rnewman]]

(In reply to Alex Davis [:adavis] from comment #5)
 
> A) Import my settings
…
> Importing settings from the default browser reduces friction for migration
> to FX. This is why we do it on desktop too.
…
> -"Import" allows you to pull settings, history and bookmarks from another
> browser you are already using.

Careful here, Alex. On Android we don't sync prefs, or a number of other customizations, so the value prop is basically bookmarks, history, and passwords.

Neither do we -- last time I checked -- import settings from other browsers.

Try to avoid the term "settings" when we talk about what's happening here. We're syncing or importing data, not anything a user would recognize as settings.


Perhaps the clearest way of phrasing what's happening is:

    Transfer your existing bookmarks and browsing history from Browser on this device

— we have a pretty good idea which browser we're importing from, and we can even say phone/tablet/device. "Transfer" or "import" and "existing" clarify that this is a one-shot import, not some kind of synchronization.

Comment 7

[Alex Davis [:adavis]]

(In reply to Richard Newman [:rnewman] from comment #6)
> (In reply to Alex Davis [:adavis] from comment #5)
>  
> > A) Import my settings
> …
> > Importing settings from the default browser reduces friction for migration
> > to FX. This is why we do it on desktop too.
> …
> > -"Import" allows you to pull settings, history and bookmarks from another
> > browser you are already using.
> 
> Careful here, Alex. On Android we don't sync prefs, or a number of other
> customizations, so the value prop is basically bookmarks, history, and
> passwords.
> 
> Neither do we -- last time I checked -- import settings from other browsers.
> 
> Try to avoid the term "settings" when we talk about what's happening here.
> We're syncing or importing data, not anything a user would recognize as
> settings.
> 
> 
> Perhaps the clearest way of phrasing what's happening is:
> 
>     Transfer your existing bookmarks and browsing history from Browser on
> this device
> 
> — we have a pretty good idea which browser we're importing from, and we can
> even say phone/tablet/device. "Transfer" or "import" and "existing" clarify
> that this is a one-shot import, not some kind of synchronization.

You're right. To avoid over-communicating to Matej (and confusing with my poor choice of words), I will let Anthony provide the guidance here. My main concern was conversion and I think I've already provided my feedback.

Comment 8

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Maybe I can provide some clarity here so Matej and I can keep focusing on the copy before we give feedback :)

(In reply to Matej Novak [:matej] from comment #4)
> Sorry, I misunderstood what was happening here. I thought it was the person
> importing their history and bookmarks from Firefox on another device, rather
> than from another browser on the same device. 
> 
> Given that Sync offers the same feature, could this be confusing to users?

Maybe. But it's also different.

Sync's value proposition is really for FxA users that are _already_ using Sync on another device. Whereas this is really about grabbing users data (regardless of whether or not they're using Sync or have an FxA) that is currently being saved in another browser, on the same device. 

> And what if the person gets a new Android phone and installs Firefox before
> even opening the native browser? It will be odd for them to see this message.

Yep. I agree, this could be kind of weird. But I think we want to validate this option through testing regardless.

And I believe Alex's point here is that this bug is focused just on getting the copy for these two versions because we'd like to validate these issues later.

> Given that we're aggressively trying to get people to sign up for an
> Account, wouldn't it be better to offer this up as another benefit of Sync
> (and therefore a reason to sign up) rather than importing their data from
> another browser?

Possibly, we're also just starting to bring testing into our workflow more on the mobile side and so this is really one of our first steps into this space. Given our current experience (the control), we'd like to see how we can add to it, so we're testing some possibilities here.

Hope that helps!

Comment 9

[Matej Novak [:matej]]

Anthony and I chatted over IRC. Right now the copy makes it feel like it's all part of the same process and that the two things are somehow related. I've tried to make them more separate and give a little more definition to what Sync does (though it's hard in this short space, especially in a way that doesn't make it confusing related to importing your data).

Have a look and let know what you think:

https://docs.google.com/document/d/1FKiwqeYfzK3dKQSAOjKx7f_X7vw9KLJuoCk1LeQNwxY/edit

P.S. I wasn't sure why the CTA for Sync was different between the two versions, but I left it that way.

Comment 10

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: Screen Shot 2015-09-01 at 9.37.20 AM.png

Thanks for the copy Matej!

I really like "Import your bookmarks and history from another browser." I think that calls out the exact value and is pretty concise. Note: instead of [browser] I used "another browser" because on Android it could really be anything depending on the OEM. Thoughts?

Attaching a screenshot of the "sync" slide here. Do you think we should talk a bit more explicitly about the value of sync? Looking at the space, we also have the capacity (at least in EN) to do so if we choose. But, I'm happy to follow your lead here.

Let me know what you think. Also, feel free to change copy in the CTA buttons :)

Comment 11

[Matej Novak [:matej]]

(In reply to Anthony Lam (:antlam) from comment #10)
> Created attachment 8655531 [details]
> Screen Shot 2015-09-01 at 9.37.20 AM.png
> 
> Thanks for the copy Matej!
> 
> I really like "Import your bookmarks and history from another browser." I
> think that calls out the exact value and is pretty concise. Note: instead of
> [browser] I used "another browser" because on Android it could really be
> anything depending on the OEM. Thoughts?

I like it, but it sounds like the user will be able to choose which browser they want to transfer their data from. Is that the case?


> Attaching a screenshot of the "sync" slide here. Do you think we should talk
> a bit more explicitly about the value of sync? Looking at the space, we also
> have the capacity (at least in EN) to do so if we choose. But, I'm happy to
> follow your lead here.

I think you're right. I made some updates to the gdoc. Have a look and let me know what you think.


> Let me know what you think. Also, feel free to change copy in the CTA
> buttons :)

Adding :rfeeley here to weigh in on the CTAs.

Ryan, can you have a look at the copy deck linked in comment 9 to see if the Sync CTAs align with the current account flow? Thanks.

Comment 12

[Ryan Feeley [:rfeeley]]

Looks good. I left one comment.

Comment 13

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

Attachment: Screen Shot 2015-09-01 at 2.09.31 PM.png

Perfect! thanks for the copy Matej, Ryan!

An update, we've decided to not test our "C" option for now. Sorry about the hassle, but feel free to ping me if you have questions. For now, we're just going with our current experience Vs. our option B.

Chenxia, I've included a mock with the updated copy for you here.

Comment 14

[Chenxia Liu [:liuche]]

Attachment: Screenshot: "Import" dialog

Okay, thanks - I'll update my strings!

Also, for the dialog that pops up when the user clicks the CTA, I'll also just use "Transfer from Firefox" for the title, and "Transfer" for the button (unless you want a different string).

I've included a screenshot here.

Comment 15

[Matej Novak [:matej]]

I see that this got assigned to me at the same time that it got resolved. Anything else needed?

Comment 16

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

(In reply to Chenxia Liu [:liuche] from comment #14)
> Created attachment 8656334 [details]
> Screenshot: "Import" dialog
> 
> Okay, thanks - I'll update my strings!
> 
> Also, for the dialog that pops up when the user clicks the CTA, I'll also
> just use "Transfer from Firefox" for the title, and "Transfer" for the
> button (unless you want a different string).
> 
> I've included a screenshot here.

WFM!

Comment 17

[Barbara Bermes [:barbara] - NI please!]

Sorry to be coming in so late, I must have missed this bug.

I'm not really sure about the copy related above the "Transfer to Firefox", the user might not know from where? From your Desktop? From "another browser" on your mobile device? Chrome? Opera? Stock? 
I think we need to be a bit more descriptive here. To be honest, if I read this as a user, I wouldn't know what it means and say skip/next.

As far as I know it only imports from Stock and not Chrome, so if we try to convince Chrome users they can import their bookmarks etc, we would be lying and probably really annoying them.

I think we should call out that the browser it's importing from is the default Android browser.

Thoughts?

Sorry again for raising this so late.

Being totally crazy, we could have another variation with the updated copy (I know that complicates things) or have this A/B test done after, what copy seems more explanatory.



I'm still trying to get data around how many people use a non-stock browser and how many browsers in general on average (I think rweiss wanted to get this for me)

Comment 18

[Matej Novak [:matej]]

How do Android users refer to the stock browser? Is "stock" a word that would make sense to them? In general we've found that people don't respond that well to "default," so it's hard to figure how to describe it in a way that would make it really clear to everyone. Any tips would be greatly appreciated.

Comment 19

[Anthony Lam (:antlam) Find me, @antlam on Slack!]

(In reply to Barbara Bermes [:bbermes] from comment #17)
> Sorry to be coming in so late, I must have missed this bug.
> 
> I'm not really sure about the copy related above the "Transfer to Firefox",
> the user might not know from where? From your Desktop? From "another
> browser" on your mobile device? Chrome? Opera? Stock? 
> I think we need to be a bit more descriptive here. To be honest, if I read
> this as a user, I wouldn't know what it means and say skip/next.
> 
> As far as I know it only imports from Stock and not Chrome, so if we try to
> convince Chrome users they can import their bookmarks etc, we would be lying
> and probably really annoying them.
> 
> I think we should call out that the browser it's importing from is the
> default Android browser.
> 
> Thoughts?

I definitely see value in being more specific. But technically we can't guarantee that this statement would be completely accurate and so I'd rather be a bit more general with our choice of words.

Also, this could change at any time if the other "browsers" decide to change the way they implement this structure on their end. Given that, this seems like a fine line to try and tread right now.

> Sorry again for raising this so late.
> 
> Being totally crazy, we could have another variation with the updated copy
> (I know that complicates things) or have this A/B test done after, what copy
> seems more explanatory.

Perhaps for our next test. I think we should focus on getting this one done first. 
 
> I'm still trying to get data around how many people use a non-stock browser
> and how many browsers in general on average (I think rweiss wanted to get
> this for me)

more data! \o/

(In reply to Matej Novak [:matej] from comment #18)
> How do Android users refer to the stock browser? Is "stock" a word that
> would make sense to them? In general we've found that people don't respond
> that well to "default," so it's hard to figure how to describe it in a way
> that would make it really clear to everyone. Any tips would be greatly
> appreciated.

That's the difficult thing here. Some apps are called "Browser", others are called "Internet". There may even be more out there. With localization, user-types, etc, it makes it really hard to know for sure.

As above, I think there's real value in being more specific. But, due to the platform and World we are working in, this is really hard to do at the moment. Plus, I don't think should be in the scope of this test. I think we can(and should) look at ways to be more specific with our copy as a part of a next test once we complete this round.

Let me know what you think Barbara!

Comment 20

[Mark Finkle (:mfinkle) (use needinfo?)]

Trying to guess the name of the existing browser will be difficult and we will fail enough to try to avoid that path.

An alternate proposal is to check and see if any existing bookmarks & history even exist _before_ we show the UI. No data could exist for several reasons:
1. User has never used the existing default browser
2. Default browser uses a storage mechanism we don't support

If there is no data to import, there is no sense showing a UI that will always fail.
If there is data to import, we can just say "Import" and not be specific. The user is left to infer we mean "from the browser you have been using".

Comment 21

[Barbara Bermes [:barbara] - NI please!]

Thanks all, all valid points. Let's wait with the A/B testing copy.

But from a user perspective I could see the following happening:

I will be confused (and even annoyed) about this feature (and probably won't use it)

a) I don't know "from where" it will import history and bookmarks
b) If I never used bookmarks and history (see mfinkle's comment), this feature won't do anything and will leave me confused.

I want to avoid disappointment for the user because they thought the feature would do something it ends up not doing (e.g. Chrome bookmarks/history can't be imported).

I think rweiss is on PTO, but I'd love to see how many Android users use the stock browser and we could then see how many fishees we can catch with this slide :)

Matej, there is no real common name for the browser based on the different device manufactures, like antlam mentioned (Internet, Browser etc.). If you think "default" is a negative, not so liked word, I wonder if we could just use "existing browser" or "standard browser".

>>If there is no data to import, there is no sense showing a UI that will always fail.
Do we know which default browser don't support this storage mechanism? If we don't filter those, I assume our A/B test will be skewed?