Survey blurb and link on /new

RESOLVED FIXED

Status

RESOLVED FIXED
a year ago
11 months ago

People

(Reporter: erenaud, Assigned: agibson)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(2 attachments)

(Reporter)

Description

a year ago
Per conversation w. Tyler and Peter we need to set up a survey to run on /new

Requirements:

- shown only to en-US
- survey must run from (asap) to, current target, first week of January 2018 (Product wants to log a baseline and then watch trends over time)

Implementation:

- display sticky window on bottom of page (as with previous surveys)

Open Questions:

- need traffic (en-US) to /new to determine x% of audience that should be shown the survey (initial, may need to adjust based on # respondents per week)
(Reporter)

Updated

a year ago
Flags: needinfo?(tdowner)
(Reporter)

Comment 1

a year ago
Tyler - please provide the link to the survey here when you have it.
Flags: needinfo?(tdowner)
Summary: Survey blurb and link on /firstrun → Survey blurb and link on /new
Just to update, I had discussed with PG and eric placing this survey on first run rather than the download page, but after re-review of the survey design requirements (part of which dictate that this survey should be taken before a user has any interaction with Firefox) the download page is necessary for the goals we want to achieve.
Eric, this survey is all good to go from our end, do you need anything else from me?
Flags: needinfo?(erenaud)
(Reporter)

Comment 5

a year ago
This document describes the work request and conditions of the survey appearing on /new - https://docs.google.com/a/mozilla.com/document/d/1nrsZ93iAWZ4H5b8cPU1XUGqpsoaIQAWuE4NW4tOXewA/edit?usp=sharing
Flags: needinfo?(erenaud)
(Reporter)

Comment 6

a year ago
Gareth - the number of respondents required has been ballparked at 100-250 per week (note the survey will be shown to en-US only visitors to /new).  

Can you please indicate the % of the en-US audience to which the survey must be shown on /new to reach that number?
Flags: needinfo?(garethcull.bugs)
(Reporter)

Comment 7

a year ago
(In reply to Eric Renaud from comment #6)
> Gareth - the number of respondents required has been ballparked at 100-250
> per week (note the survey will be shown to en-US only visitors to /new).  
> 
> Can you please indicate the % of the en-US audience to which the survey must
> be shown on /new to reach that number?

Ignore that question. 100% rollout is fine, per discussion w. Gareth
Flags: needinfo?(garethcull.bugs)
(Assignee)

Updated

a year ago
Assignee: nobody → agibson
Status: NEW → ASSIGNED
(Assignee)

Comment 8

a year ago
Created attachment 8903142 [details] [review]
GitHub pull request
(Assignee)

Comment 9

a year ago
Pull request in comment 8:

- Displays survey link for 100% of en-US audience on scene1 of /firefox/new/.
- Positions survey link top left, underneath the Firefox logo (as specified in the doc linked in Comment 5).
- Survey is behind a switch, so can be toggled on/off independently of a deployment.

Things I need to know before moving forward:

- Should this survey link be shown on scene1 (pre-download), scene2 (post-download) or both scenes?
- Is 100% of en-US audience definitely right? (in the past we usually rate-limit surveys to avoid being flooded with responses)
Flags: needinfo?(tdowner)
(Assignee)

Comment 10

a year ago
Another question:

- I used the link text "Tell us what you think about Firefox." - is that OK? (copy didn't seem to be specified)
That text looks fine to me. I don't think we need 100% rollout, we only need a few hundred responses a week, so 100% seems a bit much to me. I'll let Gareth clarify though.
Flags: needinfo?(tdowner) → needinfo?(garethcull.bugs)
(Reporter)

Comment 12

a year ago
Alex  - per the meeting this morning and discussion w. Krishna, final copy is:  "Let us know why you're choosing Firefox."
Flags: needinfo?(agibson)
(Assignee)

Comment 13

a year ago
(In reply to Eric Renaud from comment #12)
> Alex  - per the meeting this morning and discussion w. Krishna, final copy
> is:  "Let us know why you're choosing Firefox."

Done, thanks
Flags: needinfo?(agibson)
(Assignee)

Comment 14

a year ago
Ok, we're going to sample rate this survey to 25% and then monitor the results. We can always adjust if needed.
Flags: needinfo?(garethcull.bugs)
(Assignee)

Comment 15

a year ago
Ok, I pushed this to a demo for final approval:

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/

Conditions:

- Survey shows for en-US only
- Sample rate limit set to 25% of audience

I also made the following assumptions, as these are either not specified or are not quite clear in the spec:

- Survey is shown on both scene1 (pre-download) and scene2 (post-download) of /firefox/new/
- Survey is shown to desktop users only.
- Link click passes along any query params from the /firefox/new/ page to the survey URL.

These above items are how previous surveys have operated.

Tyler, let me know if these are OK or need changing.
Flags: needinfo?(tdowner)
That looks fine to me. Thank you!
Flags: needinfo?(tdowner)
(Reporter)

Comment 17

a year ago
Per Tyler's thumbs up, let's proceed please, including the previous specs you point out.
Flags: needinfo?(agibson)
(Assignee)

Comment 18

a year ago
(In reply to Eric Renaud from comment #17)
> Per Tyler's thumbs up, let's proceed please, including the previous specs
> you point out.

This is just waiting for code review
Flags: needinfo?(agibson)
(Reporter)

Comment 19

a year ago
Tyler - we'll let you know when the survey is live. please then ping me when you begin to see responses and provide data on number of respondents per day in the first few days - then we can decide if we need to modify the % of audience seeing that link.
Flags: needinfo?(tdowner)

Comment 20

a year ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/e3a4112187b139405c595f6b7cf1967f5207c8c0
[fix bug 1387573] Add user intent survey to /new

https://github.com/mozilla/bedrock/commit/309c33bb5d2d27388cfcb7b87df6e54a9c899f18
Merge pull request #5082 from alexgibson/bug-1387573-firefox-new-survey-link

[fix bug 1387573] Add user intent survey to /new

Updated

a year ago
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
(Assignee)

Comment 22

a year ago
The survey is now enabled in production, thanks all.
Thank you. I am seeing results coming in. So far it's pretty far above our target, which is fine with me, but if we want to reduce the sample to somewhere about 5-10% that would still get us enough sample.

I'll leave that decision up to you though.
Flags: needinfo?(tdowner)
We are getting significantly more responses than needed, and in an effort to minimize impact on users, can we drop this to 10%?
(Assignee)

Comment 25

a year ago
Hi Tyler, do you have enough data now for this survey? Can it be removed?
Flags: needinfo?(tdowner)
Alex,
No, the purpose of this survey is to establish a baseline leading up to and after the 57 release. It can be dropped to 10% or maybe even 5% though, as it's getting far more responses than we need.
Flags: needinfo?(tdowner)
(Assignee)

Comment 27

a year ago
Re-opening so we can adjust the sample rate. Seeing as this is going to remain on the page for a long period I suggest we go to 5%
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 29

a year ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/2896c6f02e86724e05231418d86fedb02fe710be
[fix bug 1387573] Adjust sample rate limit for firefox/new/ survey

https://github.com/mozilla/bedrock/commit/c4ff06dbdec0fefd4e23241f8fed5326a7573cff
Merge pull request #5182 from alexgibson/bug-1387573-firefox-new-survey-rate

[fix bug 1387573] Adjust sample rate limit for firefox/new/ survey

Updated

a year ago
Status: REOPENED → RESOLVED
Last Resolved: a year agoa year ago
Resolution: --- → FIXED

Updated

11 months ago
Blocks: 1424927
You need to log in before you can comment on or make changes to this bug.