1387573 - Survey blurb and link on /new

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, enhancement)

Description

[Eric Renaud [:erenaud]]

Per conversation w. Tyler and Peter we need to set up a survey to run on /new

Requirements:

- shown only to en-US
- survey must run from (asap) to, current target, first week of January 2018 (Product wants to log a baseline and then watch trends over time)

Implementation:

- display sticky window on bottom of page (as with previous surveys)

Open Questions:

- need traffic (en-US) to /new to determine x% of audience that should be shown the survey (initial, may need to adjust based on # respondents per week)

Comment 1

[Eric Renaud [:erenaud]]

Tyler - please provide the link to the survey here when you have it.

Comment 2

[Tyler Downer [He/Him]]

https://qsurvey.mozilla.com/s3/Firefox-2017-dwnbase Is the link.

Comment 3

[Tyler Downer [He/Him]]

Just to update, I had discussed with PG and eric placing this survey on first run rather than the download page, but after re-review of the survey design requirements (part of which dictate that this survey should be taken before a user has any interaction with Firefox) the download page is necessary for the goals we want to achieve.

Comment 4

[Tyler Downer [He/Him]]

Eric, this survey is all good to go from our end, do you need anything else from me?

Comment 5

[Eric Renaud [:erenaud]]

This document describes the work request and conditions of the survey appearing on /new - https://docs.google.com/a/mozilla.com/document/d/1nrsZ93iAWZ4H5b8cPU1XUGqpsoaIQAWuE4NW4tOXewA/edit?usp=sharing

Comment 6

[Eric Renaud [:erenaud]]

Gareth - the number of respondents required has been ballparked at 100-250 per week (note the survey will be shown to en-US only visitors to /new).  

Can you please indicate the % of the en-US audience to which the survey must be shown on /new to reach that number?

Comment 7

[Eric Renaud [:erenaud]]

(In reply to Eric Renaud from comment #6)
> Gareth - the number of respondents required has been ballparked at 100-250
> per week (note the survey will be shown to en-US only visitors to /new).  
> 
> Can you please indicate the % of the en-US audience to which the survey must
> be shown on /new to reach that number?

Ignore that question. 100% rollout is fine, per discussion w. Gareth

Comment 8

[Alex Gibson [:agibson]]

Attachment: GitHub pull request

Comment 9

[Alex Gibson [:agibson]]

Pull request in comment 8:

- Displays survey link for 100% of en-US audience on scene1 of /firefox/new/.
- Positions survey link top left, underneath the Firefox logo (as specified in the doc linked in Comment 5).
- Survey is behind a switch, so can be toggled on/off independently of a deployment.

Things I need to know before moving forward:

- Should this survey link be shown on scene1 (pre-download), scene2 (post-download) or both scenes?
- Is 100% of en-US audience definitely right? (in the past we usually rate-limit surveys to avoid being flooded with responses)

Comment 10

[Alex Gibson [:agibson]]

Another question:

- I used the link text "Tell us what you think about Firefox." - is that OK? (copy didn't seem to be specified)

Comment 11

[Tyler Downer [He/Him]]

That text looks fine to me. I don't think we need 100% rollout, we only need a few hundred responses a week, so 100% seems a bit much to me. I'll let Gareth clarify though.

Comment 12

[Eric Renaud [:erenaud]]

Alex  - per the meeting this morning and discussion w. Krishna, final copy is:  "Let us know why you're choosing Firefox."

Comment 13

[Alex Gibson [:agibson]]

(In reply to Eric Renaud from comment #12)
> Alex  - per the meeting this morning and discussion w. Krishna, final copy
> is:  "Let us know why you're choosing Firefox."

Done, thanks

Comment 14

[Alex Gibson [:agibson]]

Ok, we're going to sample rate this survey to 25% and then monitor the results. We can always adjust if needed.

Comment 15

[Alex Gibson [:agibson]]

Ok, I pushed this to a demo for final approval:

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/

Conditions:

- Survey shows for en-US only
- Sample rate limit set to 25% of audience

I also made the following assumptions, as these are either not specified or are not quite clear in the spec:

- Survey is shown on both scene1 (pre-download) and scene2 (post-download) of /firefox/new/
- Survey is shown to desktop users only.
- Link click passes along any query params from the /firefox/new/ page to the survey URL.

These above items are how previous surveys have operated.

Tyler, let me know if these are OK or need changing.

Comment 16

[Tyler Downer [He/Him]]

That looks fine to me. Thank you!

Comment 17

[Eric Renaud [:erenaud]]

Per Tyler's thumbs up, let's proceed please, including the previous specs you point out.

Comment 18

[Alex Gibson [:agibson]]

(In reply to Eric Renaud from comment #17)
> Per Tyler's thumbs up, let's proceed please, including the previous specs
> you point out.

This is just waiting for code review

Comment 19

[Eric Renaud [:erenaud]]

Tyler - we'll let you know when the survey is live. please then ping me when you begin to see responses and provide data on number of respondents per day in the first few days - then we can decide if we need to modify the % of audience seeing that link.

Comment 20

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/e3a4112187b139405c595f6b7cf1967f5207c8c0
[fix bug 1387573] Add user intent survey to /new

https://github.com/mozilla/bedrock/commit/309c33bb5d2d27388cfcb7b87df6e54a9c899f18
Merge pull request #5082 from alexgibson/bug-1387573-firefox-new-survey-link

[fix bug 1387573] Add user intent survey to /new

Comment 21

[[github robot]]

Commit pushed to master at https://github.com/mozmeao/www-config

https://github.com/mozmeao/www-config/commit/4212bf676f8d9cfddb0d4bcefd4bc5a6dccb5a49
[bug 1387573] Enable /firefox/new/ survey

Comment 22

[Alex Gibson [:agibson]]

The survey is now enabled in production, thanks all.

Comment 23

[Tyler Downer [He/Him]]

Thank you. I am seeing results coming in. So far it's pretty far above our target, which is fine with me, but if we want to reduce the sample to somewhere about 5-10% that would still get us enough sample.

I'll leave that decision up to you though.

Comment 24

[Tyler Downer [He/Him]]

We are getting significantly more responses than needed, and in an effort to minimize impact on users, can we drop this to 10%?

Comment 25

[Alex Gibson [:agibson]]

Hi Tyler, do you have enough data now for this survey? Can it be removed?

Comment 26

[Tyler Downer [He/Him]]

Alex,
No, the purpose of this survey is to establish a baseline leading up to and after the 57 release. It can be dropped to 10% or maybe even 5% though, as it's getting far more responses than we need.

Comment 27

[Alex Gibson [:agibson]]

Re-opening so we can adjust the sample rate. Seeing as this is going to remain on the page for a long period I suggest we go to 5%

Comment 28

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bedrock/pull/5182

Comment 29

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/2896c6f02e86724e05231418d86fedb02fe710be
[fix bug 1387573] Adjust sample rate limit for firefox/new/ survey

https://github.com/mozilla/bedrock/commit/c4ff06dbdec0fefd4e23241f8fed5326a7573cff
Merge pull request #5182 from alexgibson/bug-1387573-firefox-new-survey-rate

[fix bug 1387573] Adjust sample rate limit for firefox/new/ survey