Firefox Desktop Download: Change Content of Scene 2 as Per Results of A/B Test

RESOLVED FIXED

Status

www.mozilla.org
Pages & Content
--
enhancement
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: adavis, Assigned: jpetto)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [FXGrowth][kb=1793167], URL)

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
Created attachment 8627442 [details]
Images used for the A/B test.

The growth team ran an A/B test where Firefox for Android was promoted on scene 2.

We project that it will drive an additional 700k users to the Google Play store per month from Mozilla.org. (140% uplift)

Details of the A/B test can be found here (design/screenshots, results, specifics, code used in Optimizely, etc.):
https://mana.mozilla.org/wiki/display/FIREFOX/CTA+to+Download+Firefox+Android+on+Scene+2+of+Desktop+Download

Note that both the Google Play button and the large image below both link to the Google Play store. You'll notice in the test results that they both drive a significant amount of traffic to GP.
(Reporter)

Updated

3 years ago
Whiteboard: [FXGrowth]

Comment 1

3 years ago
We'll have to see how this change will impact localization and what we would need to do for the nearly 80 languages the pages is translated in.
(Reporter)

Comment 2

3 years ago
(In reply to Alex Davis [:adavis] from comment #0)
> Created attachment 8627442 [details]
> Images used for the A/B test.
> 
> The growth team ran an A/B test where Firefox for Android was promoted on
> scene 2.
> 
> We project that it will drive an additional 700k users to the Google Play
> store per month from Mozilla.org. (140% uplift)
> 
> Details of the A/B test can be found here (design/screenshots, results,
> specifics, code used in Optimizely, etc.):
> https://mana.mozilla.org/wiki/display/FIREFOX/
> CTA+to+Download+Firefox+Android+on+Scene+2+of+Desktop+Download
> 
> Note that both the Google Play button and the large image below both link to
> the Google Play store. You'll notice in the test results that they both
> drive a significant amount of traffic to GP.

I just wanted to add a few details:

1- Uplift is likely far greater than 140%. My test only targeted EN_US but the uplift was compared to total traffic referred to Google Play from Mozilla.org.
2- There was a typo in the English version that was tested. "you Android device" should be "your Android device".
3-In case it wasn't clear, the version B in the detailed A/B test document won.
(Assignee)

Updated

3 years ago
Assignee: nobody → jon
OS: Unspecified → All
Hardware: Unspecified → All
Whiteboard: [FXGrowth] → [FXGrowth][kb=1793167]
(Assignee)

Comment 3

3 years ago
:adavis - 

Can you procure a high-res (2x) version of the composite Android image? We like to provide high-res images where possible.

Thanks.
Flags: needinfo?(adavis)
(Reporter)

Comment 4

3 years ago
(In reply to Jon Petto [:jpetto] from comment #3)
> :adavis - 
> 
> Can you procure a high-res (2x) version of the composite Android image? We
> like to provide high-res images where possible.
> 
> Thanks.

Let me look into that. I essentially found that image online and modified it with PS. 
I'll see if I can find the original, make due with the one I have or have to find a new image.
Flags: needinfo?(adavis)
(Reporter)

Updated

3 years ago
Flags: needinfo?(adavis)
(Reporter)

Comment 5

3 years ago
Update: My version is only 1024 px so I can't re-save it. I am looking to put my hands on the high resolution version.
(Reporter)

Updated

3 years ago
Depends on: 1180981
Hey Team-

A few of thoughts/questions:

1.  Have we done any user testing or research to determine whether or not the installation instructions are on that page?  Do we need to discuss weighing the pros and cons of instructions vs the mobile download CTA?  Can we track the impact on installations of removing the instructions?

2.  I left a comment about the images in that bug. https://bugzilla.mozilla.org/show_bug.cgi?id=1180981

3.  I would love to have Matej tighten up the copy before we start localization if we are moving forward with pushing this one live.

4.  Are additional tests planned?  We should agree on the stability of this version before starting localization.

Thx,
Jen
(Assignee)

Comment 7

3 years ago
We should also think about keeping the "Your download should begin automatically. If not, click here." copy and link. It's a small bit of text that can really help a user should some ad/popup blocker prevent the auto download.

Comment 8

3 years ago
(In reply to Jennifer Bertsch [:jbertsch] from comment #6)
> 
> 3.  I would love to have Matej tighten up the copy before we start
> localization if we are moving forward with pushing this one live.

More than happy to, I'm just not sure what copy we're talking about.

Can someone share it with me?

Thanks.
(Reporter)

Comment 9

3 years ago
(In reply to Jennifer Bertsch [:jbertsch] from comment #6)
> Hey Team-
> 
> A few of thoughts/questions:
> 
> 1.  Have we done any user testing or research to determine whether or not
> the installation instructions are on that page?  Do we need to discuss
> weighing the pros and cons of instructions vs the mobile download CTA?  Can
> we track the impact on installations of removing the instructions?
> 
> 2.  I left a comment about the images in that bug.
> https://bugzilla.mozilla.org/show_bug.cgi?id=1180981
> 
> 3.  I would love to have Matej tighten up the copy before we start
> localization if we are moving forward with pushing this one live.
> 
> 4.  Are additional tests planned?  We should agree on the stability of this
> version before starting localization.
> 
> Thx,
> Jen

1. 
-The instruction are not currently personalized based on the browser or the OS so approx 80% of the time users are seeing different instructions than their actual situation. I was told this page hadn't been updated in years.
-I will see if I can get that data for the period where the test was run. Impact would have to be pretty major to outweigh the benefits.

2. Thank you so much. I replied.

3. I would ideally like to leave the copy as is. I would certainly like to work with Matej on future iterations but for now, we saw from our test that copy proposed Michael and Gareth performed 40% worst. Any change to copy should be put through another A/B test since it seems to have an enormous impact.

4. Additional tests will be planned but they will be small tweaks and it is unclear for now when we will have time to run them. If we wait until full stability of this version, we loose 23k daily English referrals to the Google Play Store (approx 40k once the 5 most popular languages are localized). That's 700k monthly or 1.2M localized to 5 languages.

@Jon Petto, great point. Do we know what % of users are impacted by a pop-up blocker at this page? If not, I can run a quick test and get data on this within just 1 day.
Flags: needinfo?(adavis)
(Reporter)

Comment 10

3 years ago
Just looping back here to address 1.2 of my last comment.

We analyzed the ratio of firstruns/downloads before, during and after the test of English speaking users. This ratio should have dropped if there was a negative impact related to the test. The reasoning being that if removing the instructions on scene 2 had a negative impact on users successfully completing install, a smaller proportion of users would have seen firstrun.

We observed no noticeable change. The ratio improved by 2-5% (depending on day) during the test when compared to the same days on the previous week but was 2-5% lower when compared to the same days the following week. 

Overall, it seems to fit within statistical deviations.
I just left a comment in bug 1180981 before finding this one. 

Sounds like some of my questions were addressed here. To summarize my comment from the other bug "By introducing an element that was not previously there, we can assume that there will be more interaction w/ that element than there was before. That in itself isn't success, so I want to be sure we're also measuring what else is being affected as a result of changing this state. That way, if we find that Desktop installs are NOT impacted negatively by introducing an Android link and Android installs go up too, that's success on both fronts. And if Desktop installs do go down, we'll know by how much and can decide next steps with that value in mind."  

Alex, for this test are you able to track these exact users through to /firstrun, or was this data from all /firstrun traffic? 

Also, are the install instructions no longer valid? Displaying instructions here was a decision made long before many of us came to Mozilla. I'd be interested to see both qualitative and quantitative data to see how useful these are. Michael Verdi and the User Research team may also have some tests/studies that may be useful.
(In reply to Alex Davis [:adavis] from comment #9)
> (In reply to Jennifer Bertsch [:jbertsch] from comment #6)
> 
> 3. I would ideally like to leave the copy as is. I would certainly like to
> work with Matej on future iterations but for now, we saw from our test that
> copy proposed Michael and Gareth performed 40% worst. Any change to copy
> should be put through another A/B test since it seems to have an enormous
> impact.

If we're using version B, we need to clean up the copy. There are two typos in it as it is.

Please remember to include me (or someone from our team) to review any tests before they go out. It doesn't make us look very good if we put things out there with mistakes in them, even if only a small number of users will see it.
One additional thought, maybe we should look at using the "send to device" widget instead of the Google Play button since the PMM team feels strongly that email/sms converts better for Desktop to Mobile conversions than the link to Google play.
(Reporter)

Comment 14

3 years ago
(In reply to Matej Novak [:matej] from comment #12)
> (In reply to Alex Davis [:adavis] from comment #9)
> > (In reply to Jennifer Bertsch [:jbertsch] from comment #6)
> > 
> > 3. I would ideally like to leave the copy as is. I would certainly like to
> > work with Matej on future iterations but for now, we saw from our test that
> > copy proposed Michael and Gareth performed 40% worst. Any change to copy
> > should be put through another A/B test since it seems to have an enormous
> > impact.
> 
> If we're using version B, we need to clean up the copy. There are two typos
> in it as it is.
> 
> Please remember to include me (or someone from our team) to review any tests
> before they go out. It doesn't make us look very good if we put things out
> there with mistakes in them, even if only a small number of users will see
> it.

We're in the process of formalizing a process for releasing A/B tests. I've added to loop in your team before launching the test to review copy.
(Reporter)

Comment 15

3 years ago
(In reply to Jennifer Bertsch [:jbertsch] from comment #13)
> One additional thought, maybe we should look at using the "send to device"
> widget instead of the Google Play button since the PMM team feels strongly
> that email/sms converts better for Desktop to Mobile conversions than the
> link to Google play.

That is a great idea. 
It's a harder A/B test that will require more than just Optimizely but I've added it to the back log for an upcoming test. It has the potential of increasing CAN/US mobile downloads even more.

We have to also consider that when iOS is released, we have to be able to open the right app store depending on which device the link is opened on. Perhaps someone has already looked into that, I'll enquire.
(Reporter)

Comment 16

3 years ago
(In reply to Holly Habstritt Gaal [:Habber] from comment #11)
> I just left a comment in bug 1180981 before finding this one. 
> 
> Sounds like some of my questions were addressed here. To summarize my
> comment from the other bug "By introducing an element that was not
> previously there, we can assume that there will be more interaction w/ that
> element than there was before. That in itself isn't success, so I want to be
> sure we're also measuring what else is being affected as a result of
> changing this state. That way, if we find that Desktop installs are NOT
> impacted negatively by introducing an Android link and Android installs go
> up too, that's success on both fronts. And if Desktop installs do go down,
> we'll know by how much and can decide next steps with that value in mind."  
> 
> Alex, for this test are you able to track these exact users through to
> /firstrun, or was this data from all /firstrun traffic? 
> 
> Also, are the install instructions no longer valid? Displaying instructions
> here was a decision made long before many of us came to Mozilla. I'd be
> interested to see both qualitative and quantitative data to see how useful
> these are. Michael Verdi and the User Research team may also have some
> tests/studies that may be useful.

I was able to compare specifically English downloads to English /firstrun which were the targeted users. As mentioned, there was no noticeable change in the firstrun/download ratio.

See quoted conversation with mverdi below on qualitative data related to install instructions. Note, Michael Verdi jointly proposed this a/b test because he also felt it safe to test. Also, on the SUMO side, Roland is very excited by this change and supported it.

Conversation with Michael Verdi: 
mverdi: "I did a bunch of user tests where they installed Firefox (but the instructions were not a focus of the test). Not one person used the instructions and for most people (chrome users) the instructions are totally wrong. This was spread out over different tests but maybe 18 people total"
adavis: "they didn't use instructions because they weren't there or because they chose to ignore them?"
mverdi: "Ignored them. They've been there for years."
(In reply to Jennifer Bertsch [:jbertsch] from comment #13)
> One additional thought, maybe we should look at using the "send to device"
> widget instead of the Google Play button since the PMM team feels strongly
> that email/sms converts better for Desktop to Mobile conversions than the
> link to Google play.

One thing to consider: the send-to-device widget is currently only translated for as many locales as we have set up to receive email/sms in their respective languages [1]. The /firefox/new page is available in many more languages than we have set up here.

[1] https://github.com/mozilla/bedrock/blob/master/bedrock/settings/base.py#L707
I think we have finished this one. Closing.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.