Per conversation w. Tyler and Peter we need to set up a survey to run on /new Requirements: - shown only to en-US - survey must run from (asap) to, current target, first week of January 2018 (Product wants to log a baseline and then watch trends over time) Implementation: - display sticky window on bottom of page (as with previous surveys) Open Questions: - need traffic (en-US) to /new to determine x% of audience that should be shown the survey (initial, may need to adjust based on # respondents per week)
Tyler - please provide the link to the survey here when you have it.
https://qsurvey.mozilla.com/s3/Firefox-2017-dwnbase Is the link.
Just to update, I had discussed with PG and eric placing this survey on first run rather than the download page, but after re-review of the survey design requirements (part of which dictate that this survey should be taken before a user has any interaction with Firefox) the download page is necessary for the goals we want to achieve.
Eric, this survey is all good to go from our end, do you need anything else from me?
This document describes the work request and conditions of the survey appearing on /new - https://docs.google.com/a/mozilla.com/document/d/1nrsZ93iAWZ4H5b8cPU1XUGqpsoaIQAWuE4NW4tOXewA/edit?usp=sharing
Gareth - the number of respondents required has been ballparked at 100-250 per week (note the survey will be shown to en-US only visitors to /new). Can you please indicate the % of the en-US audience to which the survey must be shown on /new to reach that number?
(In reply to Eric Renaud from comment #6) > Gareth - the number of respondents required has been ballparked at 100-250 > per week (note the survey will be shown to en-US only visitors to /new). > > Can you please indicate the % of the en-US audience to which the survey must > be shown on /new to reach that number? Ignore that question. 100% rollout is fine, per discussion w. Gareth
Pull request in comment 8: - Displays survey link for 100% of en-US audience on scene1 of /firefox/new/. - Positions survey link top left, underneath the Firefox logo (as specified in the doc linked in Comment 5). - Survey is behind a switch, so can be toggled on/off independently of a deployment. Things I need to know before moving forward: - Should this survey link be shown on scene1 (pre-download), scene2 (post-download) or both scenes? - Is 100% of en-US audience definitely right? (in the past we usually rate-limit surveys to avoid being flooded with responses)
Another question: - I used the link text "Tell us what you think about Firefox." - is that OK? (copy didn't seem to be specified)
That text looks fine to me. I don't think we need 100% rollout, we only need a few hundred responses a week, so 100% seems a bit much to me. I'll let Gareth clarify though.
Alex - per the meeting this morning and discussion w. Krishna, final copy is: "Let us know why you're choosing Firefox."
(In reply to Eric Renaud from comment #12) > Alex - per the meeting this morning and discussion w. Krishna, final copy > is: "Let us know why you're choosing Firefox." Done, thanks
Ok, we're going to sample rate this survey to 25% and then monitor the results. We can always adjust if needed.
Ok, I pushed this to a demo for final approval: https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/ Conditions: - Survey shows for en-US only - Sample rate limit set to 25% of audience I also made the following assumptions, as these are either not specified or are not quite clear in the spec: - Survey is shown on both scene1 (pre-download) and scene2 (post-download) of /firefox/new/ - Survey is shown to desktop users only. - Link click passes along any query params from the /firefox/new/ page to the survey URL. These above items are how previous surveys have operated. Tyler, let me know if these are OK or need changing.
That looks fine to me. Thank you!
Per Tyler's thumbs up, let's proceed please, including the previous specs you point out.
(In reply to Eric Renaud from comment #17) > Per Tyler's thumbs up, let's proceed please, including the previous specs > you point out. This is just waiting for code review
Tyler - we'll let you know when the survey is live. please then ping me when you begin to see responses and provide data on number of respondents per day in the first few days - then we can decide if we need to modify the % of audience seeing that link.
Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/e3a4112187b139405c595f6b7cf1967f5207c8c0 [fix bug 1387573] Add user intent survey to /new https://github.com/mozilla/bedrock/commit/309c33bb5d2d27388cfcb7b87df6e54a9c899f18 Merge pull request #5082 from alexgibson/bug-1387573-firefox-new-survey-link [fix bug 1387573] Add user intent survey to /new
Commit pushed to master at https://github.com/mozmeao/www-config https://github.com/mozmeao/www-config/commit/4212bf676f8d9cfddb0d4bcefd4bc5a6dccb5a49 [bug 1387573] Enable /firefox/new/ survey
The survey is now enabled in production, thanks all.
Thank you. I am seeing results coming in. So far it's pretty far above our target, which is fine with me, but if we want to reduce the sample to somewhere about 5-10% that would still get us enough sample. I'll leave that decision up to you though.
We are getting significantly more responses than needed, and in an effort to minimize impact on users, can we drop this to 10%?
Hi Tyler, do you have enough data now for this survey? Can it be removed?
Alex, No, the purpose of this survey is to establish a baseline leading up to and after the 57 release. It can be dropped to 10% or maybe even 5% though, as it's getting far more responses than we need.
Re-opening so we can adjust the sample rate. Seeing as this is going to remain on the page for a long period I suggest we go to 5%
Created attachment 8915884 [details] [review] Link to GitHub pull-request: https://github.com/mozilla/bedrock/pull/5182
Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/2896c6f02e86724e05231418d86fedb02fe710be [fix bug 1387573] Adjust sample rate limit for firefox/new/ survey https://github.com/mozilla/bedrock/commit/c4ff06dbdec0fefd4e23241f8fed5326a7573cff Merge pull request #5182 from alexgibson/bug-1387573-firefox-new-survey-rate [fix bug 1387573] Adjust sample rate limit for firefox/new/ survey