1677638 - Data review for referral survey

Status: RESOLVED FIXED

(User Research :: Project Request, task)

Description

[Alex Davis [:adavis]]

See also:
Bug 1672769 (survey)
Bug 1675890 (legal)

What questions will you answer with this data?

This is a survey that we plan to run regularly to track user satisfaction with Firefox and to understand what proportion of users are self-reporting having recommended Firefox to a friend.

Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

This will establish a baseline and we hope to improve the number of users that recommend Firefox. This is going to be a metric that the Firefox product org will eventually want to automate rather than doing a one off like this first survey.

What alternative methods did you consider to answer these questions? Why were they not sufficient?

We mostly explored other ways of asking these 6 questions but Heartbeat is the most well established way for surveying our users in product.

Can current instrumentation answer these questions?

No

List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

It's a multiple choice survey to which we'll be able to link to desktop telemetry.
Details of the survey are available here:
https://docs.google.com/document/d/19QwvWU9g8Uc6E_GqaW79VelEFpboMH-4Ygk33U1WBZg/edit#

It seems somewhere around category 2 but we're not asking questions about interactions with the product but rather with recommendations outside of the product.

Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

Again, link to survey:
https://docs.google.com/document/d/19QwvWU9g8Uc6E_GqaW79VelEFpboMH-4Ygk33U1WBZg/edit#

How long will this data be collected? Choose one of the following:

For a couple of weeks. It may be extended if we don't believe we have statistical significance.

In the future we hope to run this in an ongoing manner but we first need to figure out how to automate the survey and pipe results into a Firefox Product Metrics dashboard.

What populations will you measure?

Firefox users with profiles >3 months.

Which release channels?

Release

Which countries?

en-*

Which locales?

en-*

Any other filters? Please describe in detail below.

There will be some amount of sampling.

If this data collection is default on, what is the opt-out mechanism for users?

Just don't answer the survey.

Please provide a general description of how you will analyze this data.

SurveyGizmo will provide some insights and then we will look to our telemetry to determine if people who recommend Firefox are our Core Active users.

Where do you intend to share the results of your analysis?

With leadership. This is being tracked as part of a top level OKR being reported to Selena.

Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:

SurveyGizmo
Are you using that on the Mozilla backend? Or going directly to the third-party?

Comment 1

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

I agree that this looks like category 2 data: we anonymously and randomly surveying users about their impressions of what differentiates Firefox. As clearly optional category 2 data collection, this is fine for default-on in release.

The Heartbeat data collection mechanism is not new, and is documented in the data docs: https://docs.telemetry.mozilla.org/datasets/heartbeat.html

data-review+ from me.

Comment 2

[BugBot [:suhaib / :marco]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 3

[Andi [:andi]]

Does it ring a bell?

Comment 4

[Alex Davis [:adavis]]

This survey was run.