1234225 - Report retention

Status: RESOLVED INCOMPLETE

(Hello (Loop) :: General, defect, P1)

Comment 1

[Romain Testard [:RT]]

It sounds like we could use Hawk ID, RT to check with he server team to confirm.

Comment 2

[Romain Testard [:RT]]

Notes from a separate e-mail with Ian/Standard8/Erik related to finding a solution for a persistent enough (60 days at least) client ID:
- We cannot use the room push URL since we don't actually store the uaid for the push channel across restarts. Plus if we go to shared code, it may be harder for us to be notified when a uaid is changed.
- We actually have durable "client IDs" attached to FF profiles now (for MAU purposes); we never turn them over.

Natim, do you know if the Hawk ID would be a suitable client ID for the purpose of churn measurement?

Comment 3

[Rémy Hubscher (:natim)]

> Natim, do you know if the Hawk ID would be a suitable client ID for the purpose of churn measurement?

AFAIK it seems to fit the need yes.

Comment 4

[Romain Testard [:RT]]

RT to review with Erik at reporting meeting

Comment 5

[Romain Testard [:RT]]

Updated user story to reflect the fact we want to look at cohorts of people spanning 7 days of usage.

Comment 6

[Romain Testard [:RT]]

We have the data on the loop server already.
Katie, is this something your team could help us with?
We need the JSON output similar to what you did before (no graphing necessary, just the JSON output).

Comment 7

[Romain Testard [:RT]]

Attachment: loop_room_weekly_cohorts.json

Attaching an example JSON file for weekly user retention as an example that Jorge produced.
Monthly user retention would be another file following the same principles

Comment 8

[Romain Testard [:RT]]

Hi Katie, can you please let us know if you could help us with the creation of the JSON file per the example attached?

Comment 9

[Katie Parlante]

Romain,

Yes, we can help you, although the work will be prioritized below helping e10s and top line kpis reporting.

Adding needinfo to rweiss to review retention measure.

Comment 10

[[e-mail]]

I have one comment on the retention measures: in the user story you start with daily, weekly, and monthly measures but then only request the following reports:

Time periods to graph user retention for:
- Weekly (graph the percentage of users in each cohort that accessed Hello in the 7 days following the initial use, in the 7 days following the initial use and also the 7 days following that, .....)
- Monthly (graph the percentage of users in each cohort that had accessed Hello in the 30 days following the initial use, in the 30 days following the initial use and also the 30 days following that, .....)

Why would you also not want daily, e.g.
- Daily (graph the percentage of users in each cohort that accessed Hello the previous day following initial use).

My other comment is to verify that your definition of "usage" is the same as defined for MAU and DAU (see the data glossary here for the formal definition of "active" and "user" for other product areas [https://metrics.services.mozilla.com/firefox-kpis/])

Comment 11

[Romain Testard [:RT]]

(In reply to Rebecca Weiss from comment #10)
> I have one comment on the retention measures: in the user story you start
> with daily, weekly, and monthly measures but then only request the following
> reports:
> 
> Time periods to graph user retention for:
> - Weekly (graph the percentage of users in each cohort that accessed Hello
> in the 7 days following the initial use, in the 7 days following the initial
> use and also the 7 days following that, .....)
> - Monthly (graph the percentage of users in each cohort that had accessed
> Hello in the 30 days following the initial use, in the 30 days following the
> initial use and also the 30 days following that, .....)
> 
> Why would you also not want daily, e.g.
> - Daily (graph the percentage of users in each cohort that accessed Hello
> the previous day following initial use).

Agreed, now added back in to the user story
> 
> My other comment is to verify that your definition of "usage" is the same as
> defined for MAU and DAU (see the data glossary here for the formal
> definition of "active" and "user" for other product areas
> [https://metrics.services.mozilla.com/firefox-kpis/])

Per bug 1248569 “Active” users are the users that have joined a Room in a given period.
I see on the glossary doc that FxA, Pocket and Firefox all have different ways to define "users" and "active", it seems we should add the Hello definitions on that document?

Comment 12

[[e-mail]]

Yes, we will want to add Hello's definitions.  If you send them to me as a separate document, I'll add them to the glossary (I think the glossary is write-locked for the moment).  We won't add them until the pipeline team has provided a data set that allows to compute for those numbers that can populate the dashboard, however.  

We will also need to ensure that the logic used to compute each of those retention numbers passes muster (e.g. we should see an MVP of how the statistic is computed for a given day with real data in a notebook or something similar that can be r+).  That can be added to this bug when it's done.


(In reply to Romain Testard [:RT] from comment #11)
> (In reply to Rebecca Weiss from comment #10)
> > I have one comment on the retention measures: in the user story you start
> > with daily, weekly, and monthly measures but then only request the following
> > reports:
> > 
> > Time periods to graph user retention for:
> > - Weekly (graph the percentage of users in each cohort that accessed Hello
> > in the 7 days following the initial use, in the 7 days following the initial
> > use and also the 7 days following that, .....)
> > - Monthly (graph the percentage of users in each cohort that had accessed
> > Hello in the 30 days following the initial use, in the 30 days following the
> > initial use and also the 30 days following that, .....)
> > 
> > Why would you also not want daily, e.g.
> > - Daily (graph the percentage of users in each cohort that accessed Hello
> > the previous day following initial use).
> 
> Agreed, now added back in to the user story
> > 
> > My other comment is to verify that your definition of "usage" is the same as
> > defined for MAU and DAU (see the data glossary here for the formal
> > definition of "active" and "user" for other product areas
> > [https://metrics.services.mozilla.com/firefox-kpis/])
> 
> Per bug 1248569 “Active” users are the users that have joined a Room in a
> given period.
> I see on the glossary doc that FxA, Pocket and Firefox all have different
> ways to define "users" and "active", it seems we should add the Hello
> definitions on that document?

Comment 13

[Romain Testard [:RT]]

(In reply to Rebecca Weiss from comment #12)
> Yes, we will want to add Hello's definitions.  If you send them to me as a
> separate document, I'll add them to the glossary (I think the glossary is
> write-locked for the moment).  We won't add them until the pipeline team has
> provided a data set that allows to compute for those numbers that can
> populate the dashboard, however.  
> 
Here is the document for Firefox Hello definitions: https://docs.google.com/document/d/1bnRid9Z7c2DblGPoBJi6oBZMYWlA0ggIgD-6UTVJiHY/edit#

Comment 14

[[e-mail]]

I've copied the Active and User definitions in the "to appear" section of the data glossary.  Retention will not be added to that dashboard yet, as we have not established how many KPIs should appear on that dashboard...yet...

Once the report has been generated and we have a pipeline to these measurements, we can update the dashboard with those values (DAU, MAU, and engagement ratio).

Comment 15

[Mike Trinkala [:trink]]

Is 'initial use' the first time the a uid is seen in the data or is there some flag indicating they are a new user?  

If it is the first time the uid is seen in a data then the cohorts can change if a different slice of the data is processed (e.g. after old data is pruned). Is this acceptable?

If there is a flag 
1) What is it?  
2) We may not ever see the flag since the data is not analysed from the beginning of time and they will not show up in any cohort. Is this acceptable?

Daily clarification: graph the percentage of users in each cohort that accessed Hello the day after initial use as opposed to 'the previous day following initial use'

The MAU is specified as a 28 day window and yet retention is 30 why the difference?

Comment 16

[Katie Parlante]

(In reply to Mike Trinkala [:trink] from comment #15)
> Is 'initial use' the first time the a uid is seen in the data or is there
> some flag indicating they are a new user?  
> 
> If it is the first time the uid is seen in a data then the cohorts can
> change if a different slice of the data is processed (e.g. after old data is
> pruned). Is this acceptable?
> 
> If there is a flag 
> 1) What is it?  
> 2) We may not ever see the flag since the data is not analysed from the
> beginning of time and they will not show up in any cohort. Is this
> acceptable?
> 
> Daily clarification: graph the percentage of users in each cohort that
> accessed Hello the day after initial use as opposed to 'the previous day
> following initial use'

adding needinfo to client & server devs for best way to identify 'initial use'

Comment 17

[Rémy Hubscher (:natim)]

In a server point of view we can identified new authenticated users (new userID) and new sessions (new hawkID).

- A session can be linked to a user (FxA authenticated users)
- A session can be a unauthenticated user (in that case a session == a new unauthenticated user)

The first occurrence of a new Hawk session ID will give you a new usage of hello in a new browser (profile or user authentication).

The first occurrence of a new userID will give you a new authenticated user.

Comment 18

[Mike Trinkala [:trink]]

So described in term of the data you are providing: https://mana.mozilla.org/wiki/display/CLOUDSERVICES/Loop+Logging

the Hawk session ID is 'uid' and the userId == ??

Currently I am only looking at 'action' == "join" events do I need to look at others?

Comment 19

[Rémy Hubscher (:natim)]

According to https://github.com/mozilla-services/loop-server/blob/master/loop/middlewares.js#L75

You do not need to bother making a difference between HawkID and UserID because the server already does it for you.

The first occurrence of a new uid will give you a new user (regardless of it is authenticated or not).

Comment 20

[Mike Trinkala [:trink]]

Ok thanks

So if I start processing the data on Mar 6 every uid for that week will be considered 'new' and belong to that cohort.  This really subverts the expectation of a "new user" cohort; the cohort, as it stands, would track a group of users that all used loop within a given week  Is this acceptable?

Comment 21

[Romain Testard [:RT]]

(In reply to Mike Trinkala [:trink] from comment #15)
> Is 'initial use' the first time the a uid is seen in the data or is there
> some flag indicating they are a new user?  
> 
> If it is the first time the uid is seen in a data then the cohorts can
> change if a different slice of the data is processed (e.g. after old data is
> pruned). Is this acceptable?
> 
> If there is a flag 
> 1) What is it?  
> 2) We may not ever see the flag since the data is not analysed from the
> beginning of time and they will not show up in any cohort. Is this
> acceptable?
If I understand well this means that the retention data won't be fully accurate regarding existing users but retention will be accurate for all new users?
Given the low number of users currently and high churn this seems fine.
> 
> Daily clarification: graph the percentage of users in each cohort that
> accessed Hello the day after initial use as opposed to 'the previous day
> following initial use'
Good point, now fixed in the user story
> 
> The MAU is specified as a 28 day window and yet retention is 30 why the
> difference?

We moved the MAU to 28 days (was 30 initially) to be aligned with Mozilla's standard of tracking MAU.
If it makes more sense to track retention over 28 days (if all other retention metrics at Mozilla use 28 days) then let's do it.
Rebecca can you please confirm that 28 days is what we should look at for retention?

Comment 22

[Mike Trinkala [:trink]]

Attachment: Sample loop retention output (daily analysis)

Code: https://github.com/mozilla-services/data-pipeline/blob/c4d8159b72c158b556451aafc41f88ec2ebf62a0/hindsight/cfgs/loop/run/analysis/retention.lua

Comment 23

[Mike Trinkala [:trink]]

Who needs to review this?  I am more concerned that the business logic is what you need as for the output format it can be easily changed.

Comment 24

[Mike Trinkala [:trink]]

Attachment: loop_weekly_retention.json

Comment 25

[Mike Trinkala [:trink]]

Attachment: loop_monthly_retention.json

Comment 26

[Romain Testard [:RT]]

Thanks Mike
I NI Jorge who'll import it to our reporting dashboard and check the data

Comment 27

[jmunuera]

Mike, could we follow the format we agreed (https://bugzilla.mozilla.org/show_bug.cgi?id=1234225#c7)?

Comment 28

[Romain Testard [:RT]]

Changing the monthly retention description from 30 days to 28 days in the user story per Comment 21

Comment 29

[Mike Trinkala [:trink]]

Jorge: sure if you can provide me a specification for each type of report (a json schema(s) would be ideal as it appears there would be a different schema for each analysis interval) or you could just modify the output code directly to meet your needs.

There are several issues with the example provide which is why it wasn't used:
- It looks like you want the keys to change for each interval type i.e., "day X", "week X", and "month X" (or 4 weeks as it may now be) 
- Special casing the "week 0" to contain the date range is less than ideal (I would have actually expected Week 0 to be the total)
  - as for the date format would you still want a range for day i.e. "13 dec. 2015 - 13 dec. 2015"
- I assume "week 1" actually represent the total number of users in the cohort
- It is hard coded to fixed number of intervals and will not show long tails and the short tails end up padded with a bunch zero values

The currently attached output data has a schema that is more flexible as it doesn't have to change for each interval specification, it is more human readable since it doesn't overload interval keys and instead provides the cohort names, total number of user pre cohort and an array of interval values supporting (long/short tails)

Comment 30

[Mike Trinkala [:trink]]

The current monthly report attachment was run with configuration of 30 days: https://github.com/mozilla-services/data-pipeline/blob/master/reports/loop/run/analysis/retention_monthly.cfg#L3

It looks like the reports will be re-run when we settle on an output specification, I will set it to 28 at that point.

Comment 31

[jmunuera]

Hi Mike, we can work with your current format (is fine for us). We are working to use this structure in the product dashboard. We selected the other format because is what we have in GA for the link clickers, but it is fine to use this new format.

Comment 32

[Mike Trinkala [:trink]]

Jorge; It was quick enough to tweak the output on the re-run (these have the partial day of data error on Apr 24 too)

If this works better for you, let me know and I will upload the corrected versions in this format. FYI: since the objects are just key value pairs my json outputter doesn't order them.

So 'Week 0' is the cohort title, 'Week 1' is the total number of users in the cohort and then you have the interval values. Daily would be Day #, and monthly would be Month # (the re-run uses 28 days instead of 30)

[
...

  {
    "Week 0": "24 Apr. 2016 - 30 Apr. 2016",
    "Week 3": 186,
    "Week 1": 35017,
    "Week 4": 6,
    "Week 2": 1715
  },
  {
    "Week 3": 43,
    "Week 0": "01 May. 2016 - 07 May. 2016",
    "Week 1": 31553,
    "Week 2": 1543
  },
  {
    "Week 0": "08 May. 2016 - 14 May. 2016",
    "Week 1": 17955,
    "Week 2": 368
  }
]

Comment 33

[jmunuera]

Thanks Mike, dont worry I have already updated the dashboard with the new format you have generated.

Comment 34

[Mike Trinkala [:trink]]

Attachment: Daily report updated for Apr 24

Comment 35

[Mike Trinkala [:trink]]

Attachment: Weekly report updated for Apr 24

Comment 36

[Mike Trinkala [:trink]]

Attachment: Monthly report updated for Apr 24

Switched from a 30 day window to a 28 day window

Comment 37

[Romain Testard [:RT]]

Thanks Mike, looks like data and format are good now.
What's the best way to make it available to digest and auto-update, could we have it on https://metrics.services.mozilla.com/loop-server-dashboard/ with the other metrics?

Comment 38

[Mike Trinkala [:trink]]

I was informed this was a one-off since we cannot run it on our analysis framework (a.t.m.o).  However, it could be wired up to run as a cron job on a box that has the correct credentials and dump the results to an s3 bucket that is be published to where you need it.  If you want something official (as opposed to a cron job running on a random dev box) and in production we need to loop in Ops.  In either case it is a bit of work so a new deployment issue should be filed so we can get it into a sprint.

Comment 39

[Romain Testard [:RT]]

Do you know how the JSON files available on https://metrics.services.mozilla.com/loop-server-dashboard/ were implemented? This is good enough for us (not sure if it relies on a CRON job) and we could apply the same update/storage method?
I am surprised this was considered a one-off, this is one of the 3 Hello KPIs that we'll be reporting so we'll have to keep it updated regularly to observe the evolution as we bring changes to the product.

Comment 40

[Mike Trinkala [:trink]]

whd would be a better one to answer that

Comment 41

[Katie Parlante]

Romain,

For Q2, the priority was to make sure you have the data you need for business partner meetings.

Generally speaking, my team needs to be focusing on infrastructure and tools. We need alignment on who will be doing ongoing loop reporting work -- we should probably coordinate with Rebecca and Nick about priorities & resources.

Comment 42

[Wesley Dawson [:whd]]

The files on https://metrics.services.mozilla.com/loop-server-dashboard/ are generated by real time processing of the data using heka. Presumably we could use :trink's new filters in a similar capacity.

Comment 43

[Romain Testard [:RT]]

(In reply to Wesley Dawson [:whd] from comment #42)
> The files on https://metrics.services.mozilla.com/loop-server-dashboard/ are
> generated by real time processing of the data using heka. Presumably we
> could use :trink's new filters in a similar capacity.

That would be ideal, our data analysis happens today on Google spreadsheet and the JSON files from this page feed directly into these spreadsheets.
This JSON (retention) and the JSON resulting from bug 1248569 (engagement) would ideally be available on that page to bring us what we need.
Before I talk to Rebecca and Nick, do you know how much effort is needed to implement this?

Comment 44

[Romain Testard [:RT]]

Guys can you please help me understand the amount of effort needed here? 
We were planning on running experiments with a door hanger to bring new users to Hello and this is dependent on having a way to measure retention for these users.

Comment 45

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.