1600236 - Understand how intensity, new profile creation, new profile retention and existing profile retention correlate with MAU decline

Status: RESOLVED INACTIVE

(Data Science :: Investigation, task)

Description

[Romain Testard [:RT]]

Brief Description of the request (required):

With our focus on retention there is a need to understand the relative impact of a retention change on our reach KPI.

Assuming that the main factors impacting MAU YoY are usage intensity, new profile retention, existing profile retention and new profile creation it would be valuable to define what the equation connecting these metrics looks like.

Deliverables

• Validate that YoY MAU and YoY FxA MAU change is a factor of intensity, new profile retention, existing profile retention and new profile creation
• Define the reative importance of intensity, new profile creation, new profile retention and existing profile retention in terms of MAU impact and FxA MAU impact for a % change on these metrics will be valuable
• Define a 2020 MAU target based on previous year trends that includes matching targets for existing profile retention, new profile retention, new profile creation and intensity of use
• Current MAU metric includes users with and without attribution. It may make sense to split out attributed and non attributed users as part of this analysis since the confidence of having a reliable equation is likely much higher for attributed users than non attributed users given volatility of non attributed new profiles and non attributed retention.

Business purpose for this request (required):

Set a retention baseline for reliable 2020 retention goal definition.

Requested timelines for the request or how this fits into roadmaps or critical decisions (required):

Q4

Links to any assets (e.g Start of a PHD, BRD; any document that helps describe the project):

GUD dashboard is a good source of definitions for MAU, new user retention, existing user retention, new profile creation and intensity of use.

Name of Data Scientist (If Applicable):

Jesse

Please note if it is found that not enough information has been given this will delay the triage of this request.

Comment 1

[Jesse McCrosky]

Thanks Romain, this is definitely something we should do. I'll talk to my team about prioritization, but I'd expect some output here either the end of this month or sometime in mid-late January at the latest.

A few thoughts:

• I think we should develop both a theoretical model of how these factors relate (making some assumption will be necessary) so we understand what to expect and then also build a data-driven model based on historical data. The prediction itself has some value, but the interpretability and "story-telling" value are important.
• I wouldn't expect Intensity to play a role in MAU, as it measures "how many times a user is active within a week", which is something MAU doesn't care about. The only possible contribution would be if there are users that are some seldom active that they fall "in and out" of MAU over time. I suspect these users are an insignificant fraction, but will investigate.
• Just FYI, there is already an active workstream around developing 2020 MAU targets. I'll bring this work to their attention, but I think the methodology may already be fixed.

More updates to come.

Comment 2

[Rosanne Scholl]

As we seek evidence to support decisions about resource allocation, it will be useful to know the comparative effect on MAU of (for example) a 10% increase in rentention of new installs versus a 1% increase in retention of all users. Another example is knowing how much better or worse for MAU a 1% decrease in retention among account holders compared the same decrease among non-account holders. Without understanding this, we're left with saying "1% of a lot of users is a lot"- this 1% number is widely used but arbitrary. We have been testing for changes in flows in and out, but the KPIs are about stocks. Modelling the connection between dynamic retention and MAU gives us the link between what we currently test (in Experimenter, Browser Choice, elsewhere) and MoCo KPIs.

Comment 3

[Mike Conca [:mconca]]

I just to build on what Rosanne wrote in Comment 2 and emphasize how important this data is becoming for the organization. In particular, groups working on Firefox platform features (JavaScript, DOM, networking, WebAssembly, performance, rendering, etc.) can generally only impact retention. That is, implenting base features (usually against some web standard) doesn't necessarily lead to growth, but not implementing them clearly leads to issues, which eventually drives users away (lower retention, negative growth, decreased MAU).

Both Product and Engineering are looking to prioritize Mozilla's limited resources to maximize our Reach KPI. Projects/features that are growth-oriented (e.g. Lockwise, Secure Proxy, etc.) can generally model and forecast their expected impact to MAU, but that is much more difficult for retention-oriented features. It subsequently makes trading off competing efforts very difficult.

In the end, I want to understand, based on user research (hi, Rosanne), how many users we could potentially keep in MAU if we could improve retention via improvements in web compatibility. Based on that data, we can compare the engineering resources working on different webcompat efforts against each other as well as against potential new features, and then more easily prioritize them for the biggest impact.

Comment 4

[Jesse McCrosky]

Thanks all for the additional context. This is a priority for us. I'm off on vacation starting tomorrow, but will have at least initial results by Berlin.

Comment 5

[Jesse McCrosky]

Some very initial results are available. More work is needed, but I go on vacation tomorrow, so wanted to leave something for those that are curious to read.

https://docs.google.com/document/d/1W-6m6hGrO9zWtcbfVgkxqjYEb0kNMsIQX2BFrm0JlV4/

Comment 6

[Jesse McCrosky]

A complete first draft is now available: https://docs.google.com/document/d/1W-6m6hGrO9zWtcbfVgkxqjYEb0kNMsIQX2BFrm0JlV4/

Comment 8

[Jess Mokrzecki [:jmok]]

Work for the DS team is now tracked in Jira. You can search with the Data Science Jira project for the corresponding ticket.