Survey blurb and link on /new

RESOLVED FIXED

Status

www.mozilla.org
Pages & Content
RESOLVED FIXED
4 months ago
2 months ago

People

(Reporter: Eric Renaud, Assigned: agibson)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(2 attachments)

(Reporter)

Description

4 months ago
Per conversation w. Tyler and Peter we need to set up a survey to run on /new

Requirements:

- shown only to en-US
- survey must run from (asap) to, current target, first week of January 2018 (Product wants to log a baseline and then watch trends over time)

Implementation:

- display sticky window on bottom of page (as with previous surveys)

Open Questions:

- need traffic (en-US) to /new to determine x% of audience that should be shown the survey (initial, may need to adjust based on # respondents per week)
(Reporter)

Updated

4 months ago
Flags: needinfo?(tdowner)
(Reporter)

Comment 1

4 months ago
Tyler - please provide the link to the survey here when you have it.

Updated

4 months ago
Flags: needinfo?(tdowner)
Summary: Survey blurb and link on /firstrun → Survey blurb and link on /new

Comment 2

4 months ago
https://qsurvey.mozilla.com/s3/Firefox-2017-dwnbase Is the link.

Comment 3

4 months ago
Just to update, I had discussed with PG and eric placing this survey on first run rather than the download page, but after re-review of the survey design requirements (part of which dictate that this survey should be taken before a user has any interaction with Firefox) the download page is necessary for the goals we want to achieve.

Comment 4

4 months ago
Eric, this survey is all good to go from our end, do you need anything else from me?
Flags: needinfo?(erenaud)
(Reporter)

Comment 5

4 months ago
This document describes the work request and conditions of the survey appearing on /new - https://docs.google.com/a/mozilla.com/document/d/1nrsZ93iAWZ4H5b8cPU1XUGqpsoaIQAWuE4NW4tOXewA/edit?usp=sharing
Flags: needinfo?(erenaud)
(Reporter)

Comment 6

4 months ago
Gareth - the number of respondents required has been ballparked at 100-250 per week (note the survey will be shown to en-US only visitors to /new).  

Can you please indicate the % of the en-US audience to which the survey must be shown on /new to reach that number?
Flags: needinfo?(garethcull.bugs)
(Reporter)

Comment 7

3 months ago
(In reply to Eric Renaud from comment #6)
> Gareth - the number of respondents required has been ballparked at 100-250
> per week (note the survey will be shown to en-US only visitors to /new).  
> 
> Can you please indicate the % of the en-US audience to which the survey must
> be shown on /new to reach that number?

Ignore that question. 100% rollout is fine, per discussion w. Gareth
Flags: needinfo?(garethcull.bugs)
(Assignee)

Updated

3 months ago
Assignee: nobody → agibson
Status: NEW → ASSIGNED
(Assignee)

Comment 8

3 months ago
Created attachment 8903142 [details] [review]
GitHub pull request
(Assignee)

Comment 9

3 months ago
Pull request in comment 8:

- Displays survey link for 100% of en-US audience on scene1 of /firefox/new/.
- Positions survey link top left, underneath the Firefox logo (as specified in the doc linked in Comment 5).
- Survey is behind a switch, so can be toggled on/off independently of a deployment.

Things I need to know before moving forward:

- Should this survey link be shown on scene1 (pre-download), scene2 (post-download) or both scenes?
- Is 100% of en-US audience definitely right? (in the past we usually rate-limit surveys to avoid being flooded with responses)
Flags: needinfo?(tdowner)
(Assignee)

Comment 10

3 months ago
Another question:

- I used the link text "Tell us what you think about Firefox." - is that OK? (copy didn't seem to be specified)
That text looks fine to me. I don't think we need 100% rollout, we only need a few hundred responses a week, so 100% seems a bit much to me. I'll let Gareth clarify though.
Flags: needinfo?(tdowner) → needinfo?(garethcull.bugs)
(Reporter)

Comment 12

3 months ago
Alex  - per the meeting this morning and discussion w. Krishna, final copy is:  "Let us know why you're choosing Firefox."
Flags: needinfo?(agibson)
(Assignee)

Comment 13

3 months ago
(In reply to Eric Renaud from comment #12)
> Alex  - per the meeting this morning and discussion w. Krishna, final copy
> is:  "Let us know why you're choosing Firefox."

Done, thanks
Flags: needinfo?(agibson)
(Assignee)

Comment 14

3 months ago
Ok, we're going to sample rate this survey to 25% and then monitor the results. We can always adjust if needed.
Flags: needinfo?(garethcull.bugs)
(Assignee)

Comment 15

3 months ago
Ok, I pushed this to a demo for final approval:

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/

Conditions:

- Survey shows for en-US only
- Sample rate limit set to 25% of audience

I also made the following assumptions, as these are either not specified or are not quite clear in the spec:

- Survey is shown on both scene1 (pre-download) and scene2 (post-download) of /firefox/new/
- Survey is shown to desktop users only.
- Link click passes along any query params from the /firefox/new/ page to the survey URL.

These above items are how previous surveys have operated.

Tyler, let me know if these are OK or need changing.
Flags: needinfo?(tdowner)
That looks fine to me. Thank you!
Flags: needinfo?(tdowner)
(Reporter)

Comment 17

3 months ago
Per Tyler's thumbs up, let's proceed please, including the previous specs you point out.
Flags: needinfo?(agibson)
(Assignee)

Comment 18

3 months ago
(In reply to Eric Renaud from comment #17)
> Per Tyler's thumbs up, let's proceed please, including the previous specs
> you point out.

This is just waiting for code review
Flags: needinfo?(agibson)
(Reporter)

Comment 19

3 months ago
Tyler - we'll let you know when the survey is live. please then ping me when you begin to see responses and provide data on number of respondents per day in the first few days - then we can decide if we need to modify the % of audience seeing that link.
Flags: needinfo?(tdowner)

Comment 20

3 months ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/e3a4112187b139405c595f6b7cf1967f5207c8c0
[fix bug 1387573] Add user intent survey to /new

https://github.com/mozilla/bedrock/commit/309c33bb5d2d27388cfcb7b87df6e54a9c899f18
Merge pull request #5082 from alexgibson/bug-1387573-firefox-new-survey-link

[fix bug 1387573] Add user intent survey to /new

Updated

3 months ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → FIXED

Comment 21

3 months ago
Commit pushed to master at https://github.com/mozmeao/www-config

https://github.com/mozmeao/www-config/commit/4212bf676f8d9cfddb0d4bcefd4bc5a6dccb5a49
[bug 1387573] Enable /firefox/new/ survey
(Assignee)

Comment 22

3 months ago
The survey is now enabled in production, thanks all.
Thank you. I am seeing results coming in. So far it's pretty far above our target, which is fine with me, but if we want to reduce the sample to somewhere about 5-10% that would still get us enough sample.

I'll leave that decision up to you though.
Flags: needinfo?(tdowner)
We are getting significantly more responses than needed, and in an effort to minimize impact on users, can we drop this to 10%?
(Assignee)

Comment 25

2 months ago
Hi Tyler, do you have enough data now for this survey? Can it be removed?
Flags: needinfo?(tdowner)
Alex,
No, the purpose of this survey is to establish a baseline leading up to and after the 57 release. It can be dropped to 10% or maybe even 5% though, as it's getting far more responses than we need.
Flags: needinfo?(tdowner)
(Assignee)

Comment 27

2 months ago
Re-opening so we can adjust the sample rate. Seeing as this is going to remain on the page for a long period I suggest we go to 5%
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 8915884 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/bedrock/pull/5182

Comment 29

2 months ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/2896c6f02e86724e05231418d86fedb02fe710be
[fix bug 1387573] Adjust sample rate limit for firefox/new/ survey

https://github.com/mozilla/bedrock/commit/c4ff06dbdec0fefd4e23241f8fed5326a7573cff
Merge pull request #5182 from alexgibson/bug-1387573-firefox-new-survey-rate

[fix bug 1387573] Adjust sample rate limit for firefox/new/ survey

Updated

2 months ago
Status: REOPENED → RESOLVED
Last Resolved: 3 months ago2 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.