Closed Bug 1550176 Opened 6 years ago Closed 6 years ago

Heartbeat survey request for Trailhead Onboarding

Categories

(User Research :: Project Request, task, P1)

Product:
User Research
Component:
Project Request
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: tspurway, Assigned: rscholl)

References

Details

Request Description
We would like to gauge the effect of the new Trailhead First Run experience on Firefox Desktop in the experiment we will run as part of 67.0.5 starting June 4, 2019.

The experiment will test two facets of the onboarding experience: the messaging and the UX flow. We are testing variants of classic and proposed messaging as well as the lack of extra UX steps.

We would like to attach a Heartbeat survey to the experiment.

We would like to learn the following:

  • are the users intimidated, bored, or otherwise "turned off" by the additional steps they are asked to take as part of the onboarding process
  • did the users notice the messages presented? did they understand them? did they find them useful? were they relevant?
  • did the user feel like the were required to leave their email address in order to use Firefox?
  • were there too many steps to "start browsing"?

I know we have done some other surveys with the New Tab page and Onboarding. Maybe we could recycle some of those questions from the previous studies to provide a basis for comparison?

Deadline
The experiment launches with Trailhead on June 4th, and we would like to launch the survey immediately (within 15 minutes) after the onboarding experience of the participants.

Priority Level
(1 = High 2 = Medium, 3 = Low): i'm going to say 1.5 :-)

Priority Level Description
(Please provide information on how you chose the priority level above, which product management OKR(s) this work supports, or any other details that will help us triage and prioritize this request.)

This study will directly influence decisions we will make to optimize the Firefox onboarding experience both in the Trailhead initiative in June and going forward. Users who download Firefox are a crucial cohort - if we can help new users become long term users, we may solve our critical market share issues.

Supporting Information
(Links to any assets or previous research relevant to this project.)

Experiment: https://bugzilla.mozilla.org/show_bug.cgi?id=1548072
Tech Docs: https://docs.google.com/document/d/1t3dDeaAKwLpH39a2LZ7xYzX4n17w-IGiTUpk0xU9LdY/edit
Prd:https://docs.google.com/document/d/1OaZWbRSl5jtXWatxfkPpAuhKK7Kkep5KAm_OtfNJogE/edit

Blocks: 1548072

Meeting notes re targeting feasibility and concepts to measure in survey: https://docs.google.com/document/d/1rCyVg62xRFC3VkJZ3pRGmpg1WFNEI0SDdAPlZt4-mZc/edit?usp=sharing

Josh and Rosanne are working on this already and are planned to equally share different responsibilities. This bug should be triaged to one of us.

Current proposed schedule and responsibilities:
Done: May 7: scoping and feasibility meeting (R, J)
Done: May 9: targeting and concepts to measure meeting (R, J)
Done: May 9: write a fast survey for test (R)
Done: May 9: launch trial survey to see if plan A works (J with Mythmon)
May 17: analysis distributed on whether timing for Plan A test looks good (J)
May 10-20- finalize the concepts we want to measure/ draft & circulate survey (R)
June 4- finalize targeting parameters (J)
June 4 - last date to finalize survey in online context (R)
June 4- users begin having the new experience
June 4 - begin issuing survey invites and end one week later (J)
One to two weeks after that-draft report distributed (R, J)

Assignee: nobody → rscholl
Priority: -- → P1

Sent first draft of survey to stakeholders; requested edits. https://docs.google.com/document/d/1D05Hp3Y_Y3_Z8mcckjD0Igcz7QyiqEdLtKV3zCpKVrA/edit?usp=sharing

Status: NEW → ASSIGNED

Put survey in-context (minus responsiveness to variant):
https://app.surveygizmo.com/builder/build/id/5028620
and sent to stakeholders for review:
https://qsurvey.mozilla.com/collab/2019-Firefox-introduction-survey

Jim - can you give the official thumbs up on the survey? Link here to review: https://qsurvey.mozilla.com/collab/2019-Firefox-introduction-survey
(If you have edits next week, Josh, we'll need your help to get the edits made)

Mythmon - please confirm when you've got the recipes ready to launch (by 5/30).

Josh - please confirm if you're launching the recipes on 5/30 or if you want mythmon to step in.

Josh/mythmon - when the recipes are live (5/30), please comment in this bug.

Thanks all! \o/

Flags: needinfo?(jimthomas)

The survey is now responsive to the variant and is ready for final review.
Here are the links for your review and for the recipes:
https://qsurvey.mozilla.com/s3/2019-Firefox-introduction-survey/?b=sync
https://qsurvey.mozilla.com/s3/2019-Firefox-introduction-survey/?b=join
https://qsurvey.mozilla.com/s3/2019-Firefox-introduction-survey/?b=control
https://qsurvey.mozilla.com/s3/2019-Firefox-introduction-survey/?b=nofirstrun
https://qsurvey.mozilla.com/s3/2019-Firefox-introduction-survey/?b=cards
The first three have an additional page between the first and last pages, with a screenshot that is responsive to the variant.
I have incorporated edits from mardak and jgaunt and added a final question that was requested by jdavidson for a related project.
While I'm out next week, jgaunt (builder link above) can make any additional changes if necessary, or I can on June 3.

This looks great!

Minor feedback now that I see it in action... this option on page 3:

Something Else: Some other promise is what I want Firefox to give me.

is a little hard to understand when it's presented in a random order and in the context of ranking. Can we remove that? If not, it's ok to go as-is also.

Flags: needinfo?(jimthomas)

For the heartbeat recipes, I still need a few things:

  • Message: The message to include in the heartbeat
  • Engagement button label: The text to show on the button that opens they survey
  • Learn more URL: A page that has more details about the request (this is not the survey)
  • Learn for Message: Text for the learn more link (this is often just "Learn more")
  • Thanks Message: Text that is shown briefly on the bar after the user opens the survey. This is often "Thank you"

These could be all the same for all 5 surveys, but could also vary per treatment if we want.

Flags: needinfo?(tspurway)

I've got the details I needed via Slack.

Flags: needinfo?(tspurway)

I've queued up 5 recipes for this survey, and requested approval for them. I used the same "second run" targeting used in recipe 770, which we used to validate this approach.

Josh, can you review these?

Flags: needinfo?(jgaunt)

Recipes approved and published.

Funnel tracking available here (starting next week): https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/126190

Flags: needinfo?(jgaunt)

Deck that summarizes and contextualizes results: https://docs.google.com/presentation/d/1uSel0v4_TVfagsA_GTCW3C1z1gJtdNPemjYhzWzkdO8/edit?ts=5d043504#slide=id.p Links to survey data inside

Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.