932572 - [Survey] First Run Users

Status: RESOLVED FIXED

(www.mozilla.org :: General, defect)

Description

[Laura Forrest]

It seems like we have some open questions when it comes to figuring out who the users are that hit the First Run page. 

A hypothesis is that maybe a certain percentage of those visits come from users who are downloading a fresh build, or downloading to a new machine for the first time. Let's gather insights via a survey to answer the question of "How many of these visitors are true first-time Firefox users"

Possible survey design:
Q: Have you ever downloaded and used Firefox before?
A: Yes/No

If Yes

Q: Why are you downloading Firefox again?
A1: I got a new computer and want it on this one, too
A2: I want a fresh build of Firefox
A3: I want the most recent version of Firefox
A4: Other - Write In

Other hypothesis or survey designs are welcome.

Comment 1

[hhabstritt]

As for implementation, I can create surveys for this using Survey Gizmo and work with Craig to embed in a nice style that is triggered from the /firstrun page.

Comment 2

[Chris More [:cmore]]

Do we want to show this survey for all visitors or only a sample? Do we want to start with en-US only then do another locale later?

Comment 3

[Laura Forrest]

(In reply to Chris More [:cmore] from comment #2)
> Do we want to show this survey for all visitors or only a sample? 

I've found that 1% is a good sample size since these pages get so much traffic. We did this on the What's New page ages ago, here's Copy, Code, and Sampling info that may or may not help this time. 

https://bugzilla.mozilla.org/show_bug.cgi?id=660996#c33  

Do we want
> to start with en-US only then do another locale later?

Let's start with en-US. I don't want to add project scope, I think it's wise to run this survey in an Emerging Market too, like Russia, India, Indonesia, or Brazil since overall behavior could be very different in markets where our product is mature and has high awareness (US). 

The hypothesis there is that a greater percentage are downloading for the very first time in emerging markets vs. checking for updates/downloading a "fresh" build in mature markets.

Comment 4

[Chris More [:cmore]]

Laura: %1 should be fine as that is about 5,000+ per day.

Craig:  We could do the 1% sample via JavaScript like:

var surveyProbability = 0.01;
showSurvey = (Math.random() < surveyProbability) ? 'yes' : 'no';

An iframe to SurveyGizmo was done here:

http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/en-US/products/download.html?revision=96546&view=markup&pathrev=96546

Thanks all!

Comment 5

[Laura Forrest]

Great! When will this launch?

Comment 6

[Chris More [:cmore]]

Holly: Can you create the first pass at the survey using the information here and post a link so we can review?

Comment 7

[Holly Habstritt Gaal [:Habber]]

(In reply to Chris More [:cmore] from comment #6)
> Holly: Can you create the first pass at the survey using the information
> here and post a link so we can review?

Yep. No problem. I can post something here next week.

Comment 8

[Holly Habstritt Gaal [:Habber]]

Hello. This form is ready to go and you can preview it here: http://www.surveygizmo.com/s3/1450052/First-Run-has-the-user-downloaded-Firefox-before

If the user selects 'no' in the first question, they receive a Thank You. If they select 'yes' they get one follow-up question. Let me know if you have feedback on the copy.

The questions in the form are now mandatory because I expect that this will be styled and displayed in a modal where the user can opt out. I can work with Craig on how this is displayed.


I can provide code for this in the following ways. Just let me know what you need.
- Javascript embed
- Website pop-up
- Static HTML embed
- iFrame embed

I see Craig isn't cc'd on this yet. Jen or Chris, is he confirmed as the dev working with us to implement this?

Comment 9

[Jennifer Bertsch [:jbertsch]]

Hi Mike and Jon-

Do you think we could work on this one in the first half of December?

Thx,
Jen

Comment 10

[Chris More [:cmore]]

(In reply to Holly Habstritt [:Habber] from comment #8)
> Hello. This form is ready to go and you can preview it here:
> http://www.surveygizmo.com/s3/1450052/First-Run-has-the-user-downloaded-
> Firefox-before
> 
> If the user selects 'no' in the first question, they receive a Thank You. If
> they select 'yes' they get one follow-up question. Let me know if you have
> feedback on the copy.
> 
> The questions in the form are now mandatory because I expect that this will
> be styled and displayed in a modal where the user can opt out. I can work
> with Craig on how this is displayed.
> 
> 
> I can provide code for this in the following ways. Just let me know what you
> need.
> - Javascript embed
> - Website pop-up
> - Static HTML embed
> - iFrame embed
> 
> I see Craig isn't cc'd on this yet. Jen or Chris, is he confirmed as the dev
> working with us to implement this?

I like the survey!

Malexis: Let us know if there is any availability between now and the end of the year to put get the survey in the en-US version of the firstrun page.

Comment 11

[Laura Forrest]

Adding John Jensen to keep him informed. He shared a stat with me earlier today that about 48% of Fx downloads are pave-overs (meaning people already have a Fx profile on their machine). This survey will hopefully help unearth why this happens at such a strikingly high rate.

Comment 12

[Holly Habstritt Gaal [:Habber]]

Laura: We have 2 questions for you.

- What percentage of users should we show this to?

- Should the form be displayed automatically over the page (ie: in a lightbox) or should we add a link on the first run page for the user to expand the survey on their own.

Comment 13

[Holly Habstritt Gaal [:Habber]]

I see in comment 4 our first question has been answered.

Comment 14

[Laura Forrest]

(In reply to Holly Habstritt [:Habber] from comment #12)
> Laura: We have 2 questions for you.
> 
> - What percentage of users should we show this to?
> 
> - Should the form be displayed automatically over the page (ie: in a
> lightbox) or should we add a link on the first run page for the user to
> expand the survey on their own.

Whatever you think is the better user experience. Perhaps the link, since it's less invasive.

Comment 15

[Holly Habstritt Gaal [:Habber]]

Great. We'll go with putting the link to trigger the form in page and if that doesn't result in enough responses for us we can always try the immediate light box. 


(In reply to Laura Forrest from comment #14)
> (In reply to Holly Habstritt [:Habber] from comment #12)
> > Laura: We have 2 questions for you.
> > 
> > - What percentage of users should we show this to?
> > 
> > - Should the form be displayed automatically over the page (ie: in a
> > lightbox) or should we add a link on the first run page for the user to
> > expand the survey on their own.
> 
> Whatever you think is the better user experience. Perhaps the link, since
> it's less invasive.

Comment 16

[Craig Cook (:craigcook)]

Any thoughts on copy for the button? Maybe "<button>Take a brief survey</button> and help us make Firefox better."?

Comment 17

[Laura Forrest]

(In reply to Craig Cook (:craigcook) from comment #16)
> Any thoughts on copy for the button? Maybe "<button>Take a brief
> survey</button> and help us make Firefox better."?

Sounds good to me.

Comment 18

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/57dec44033e1e715fcddd737145ae63639b42c65
Fix bug 932572 - add download survey to first run page

https://github.com/mozilla/bedrock/commit/401e234efb5097adc72229d203feda0946cc86ba
Merge pull request #1532 from craigcook/bug-932572-first-run-survey

Fix bug 932572 - add download survey to first run page

Comment 19

[raymond [:retornam] (inactive)]

fixed on stage https://www.allizom.org/en-US/firefox/firstrun/

Comment 20

[Holly Habstritt Gaal [:Habber]]

Here is a high level report that displays percentages for each answer, free form answers from the "other" field, and what locations we received responses from:
http://cl.ly/092G1V0S070u

This report gives us a better view of which countries are more likely to download the browser multiple times and why: 
http://cl.ly/1N1l010e190V

It is interesting to see countries like Indonesia, India, Pakistan, and Philippines stick out as non-US countries that are downloading the browser again. These countries also fall into a list of countries that have low download speeds, but high amount of attempts to download the product. Our research in Indonesia has shown us that silent updates are not working well when a user has an unreliable connection and this could be why we see users from these countries getting a fresh copy of the browser.

Chris, do you have any thoughts about how this could relate to your download size tests?

Laura, is there anything else you'd like to learn from users reaching the First Run page?


We have received enough responses and will remove the survey.

Comment 21

[Chris More [:cmore]]

The following are the focus countries that I mentioned in my download tests (higher downloads are first):

India
Indonesia
Pakistan
Egypt
Iran
Bangladesh
Nigeria
Ghana
Kenya

These are the people with the lowest success rates and still had a decent amount of download attempts. I think we would need a second survey to understand what we could do differently for those people. It would be interesting to do a survey one of the top three countries on the installer-help page to see if we can ask questions on what we could do differently to serve them better.

Comment 22

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

Clearing the needinfo request since it is pending on an account that cannot answer (a bot, a component watcher, or a disabled account).

For more information, please visit BugBot documentation.