Closed Bug 1208416 Opened 5 years ago Closed 4 years ago

Report on Hello MAUs per new MAU definition

Categories

(Hello (Loop) :: Client, defect, P1)

defect

Tracking

(firefox46 fixed, firefox47 fixed)

RESOLVED FIXED
mozilla47
Iteration:
47.3 - Mar 7
Tracking Status
firefox46 --- fixed
firefox47 --- fixed

People

(Reporter: RT, Assigned: mancas)

References

Details

(Whiteboard: [btpp-follow-up-2016-03-08])

User Story

We agreed on the new MAU definition as follows:
1 Firefox users who have activated either the conversation window or the Hello panel in the last 30 days
2 Non Firefox users (or Firefox users of the standalone link clicker website) who have reached a Hello URL in the last 30 days

2 will be tracked using GA (already instrumented

We although need to get metrics regarding "1" and this implies counting users per 30 days periods who have either opened the panel or joined a Hello conversation directly.
Since the ability to join a Hello conversation directly will come later (related to clicker parity) we can focus on counting unique users per 30 day period who have opened the Hello panel.

Acceptance criteria:
- count unique users per 30 day period who have opened the Hello panel.

FHR sounds like an appropriate vehicle for this.

Attachments

(3 files, 3 obsolete files)

No description provided.
User Story: (updated)
Rank: 28
Priority: -- → P2
User Story: (updated)
I've added Bsmedberg to this.  He or one of the folks he delegates to should look at the FHR proposal.
Phases of this:
* Determine the requirements (done).
* Determine what the user benefit is. Per both our own policy and various legal requirements, we can't collect this data on an opt-out basis without providing a user benefit. It is typically up to the Hello team to articulate the user value, although Marshall and I can help with this. If the data collection is via a service, it may be as simple as "we need to collect this data on our server in order to provide the service".
* Determine what measures satisfy the requirements - this should be done by the Hello team, consulting with the data collection team to make sure that we are respecting our privacy principles. A key decision here is whether we're going to collect this from the client or as part of some server-based system.
* When the Hello team has implemented the final metrics, I or a peer will review the data and matching documentation for final data collection review.
* The Hello team will be responsible for implementing the dashboards that track these metrics over time. Katie's measurement team can assist with training and we have devops support to monitor and support dashboards after they are built.
* There must be a person responsible for monitoring the data over time (presumably Romain, but this should be explicit).
Rank: 28 → 25
Rank: 25 → 28
User value and proposed data collection:

The user value we are providing through this data collection is the ability for the Hello team to understand engagement with the product, and use that information to make the product more useful or engaging, or determine if the product doesn't have sufficient value to users.

The data we propose collecting: we want to measure if and how people engage with the Hello product.  There are two levels of engagement: opening the panel, and interacting with the panel (starting a conversation, interacting with the room list, etc).  We also want to measure churn/retention, so we want to know how many unique users open or engage with the panel, and how often.  We'd like to be able to create reports with counts of the general categories (over 30 day periods):

1. Users who open the panel but do not engage
2. Users who engage once with the panel, but do not return
3. Users who engage multiple times with the panel

"Engagement" means the user clicks on at least one active element in the panel.
Blocks: 1234225
Mike: please break down this bug, I'd like to talk about it on Tuesday's standup
Rank: 28 → 24
Flags: needinfo?(mdeboer)
(In reply to Ian Bicking (:ianb) from comment #4)
> Mike: please break down this bug, I'd like to talk about it on Tuesday's
> standup

The story in comment 3 doesn't seem to be in-sync with the US. I'd like discuss moving the '30 day' window to the reporting end to filter down, instead of making it an implementation detail. That way we can simply track (in FHR):
1. Panel open frequency.
2. Clicked a room entry
 a. to open
 b. to share via email/ copy
 c. to delete
2. Chat open frequency
Flags: needinfo?(rtestard)
Flags: needinfo?(mdeboer)
Flags: needinfo?(ianb)
(In reply to Mike de Boer [:mikedeboer] from comment #5)

> discuss moving the '30 day' window to the reporting end to filter down,
> instead of making it an implementation detail. That way we can simply track
> (in FHR):
> 1. Panel open frequency.
> 2. Clicked a room entry
>  a. to open
>  b. to share via email/ copy
>  c. to delete
> 2. Chat open frequency
Makes total sense, let's do that. I specified it only to highlight the fact we need to get the 30 day window eventually, it's indeed better if we can get that on the reporting end so long as you provide the right timestamps to allow that.
Flags: needinfo?(rtestard)
Assignee: nobody → b.mcb
Comment on attachment 8717809 [details] [review]
[loop] mancas:bug1208416 > mozilla:master

Asking you for feedback because we talked yesterday about this bug, but feel free to assign the review to other person.

Thanks!
Attachment #8717809 - Flags: feedback?(mdeboer)
Attachment #8717809 - Flags: feedback?(mdeboer) → feedback+
Comment on attachment 8717809 [details] [review]
[loop] mancas:bug1208416 > mozilla:master

Comments addressed Mike
Attachment #8717809 - Flags: feedback+ → review?(mdeboer)
Attached patch MC_Patch (obsolete) — Splinter Review
Comment on attachment 8717809 [details] [review]
[loop] mancas:bug1208416 > mozilla:master

Aaaaaalmost there!!
Attachment #8717809 - Flags: review?(mdeboer) → review-
Comment on attachment 8718297 [details] [diff] [review]
MC_Patch

I think we haven't followed up with some of Benjamin's questions, so:

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #2)
> * Determine what the user benefit is. Per both our own policy and various
> legal requirements, we can't collect this data on an opt-out basis without
> providing a user benefit. It is typically up to the Hello team to articulate
> the user value, although Marshall and I can help with this. If the data
> collection is via a service, it may be as simple as "we need to collect this
> data on our server in order to provide the service".

The MAU gives us a baseline KPI for Hello, which allows us to measure the effectiveness and utility of the product for users, and see how changes we make in Hello increase (or don't) that value.

> * Determine what measures satisfy the requirements - this should be done by
> the Hello team, consulting with the data collection team to make sure that
> we are respecting our privacy principles. A key decision here is whether
> we're going to collect this from the client or as part of some server-based
> system.

To get a complete view of the funnel we are measuring, we are counting all activity that touches Hello, from the point where someone clicks on the Hello icon.  

> * When the Hello team has implemented the final metrics, I or a peer will
> review the data and matching documentation for final data collection review.

Besides this comment, is there someplace I should document this?  (Or someplace where I should copy the justification from this comemnt?)

> * The Hello team will be responsible for implementing the dashboards that
> track these metrics over time. Katie's measurement team can assist with
> training and we have devops support to monitor and support dashboards after
> they are built.
> * There must be a person responsible for monitoring the data over time
> (presumably Romain, but this should be explicit).

Romain Testard will be responsible.
Flags: needinfo?(ianb)
Attachment #8718297 - Flags: review?(benjamin)
Attachment #8717809 - Flags: review- → review?(mdeboer)
Attachment #8717809 - Flags: review?(mdeboer) → review+
Blocks: 1248602
LOOP_MAU is not a good name for this. I recommend LOOP_ACTIVITY_COUNTER.

Since histograms are self-documenting, if this is the only data collection you're doing the change to Histograms.json should be sufficient documentation.

I'd really like you to be more specific about the user value: "measure the effectiveness and utility of the product for users" is still pretty vague. What specific product improvements will be driven by this data? Note that our own KPIs or understanding of users are not considered user value until we can tie them to more specific product improvements that will be driven by this data.

You've said that Romain will be monitoring the dashboards... who will be building them and is there already a specification of what they will contain?

Also I'll note that you have this expiring in Firefox 50, which is relatively quickly. I expect you want this to be on for at least 6 months in release, which would be at least Firefox 51.
Flags: needinfo?(rtestard)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #14)
> LOOP_MAU is not a good name for this. I recommend LOOP_ACTIVITY_COUNTER.
> 
> Since histograms are self-documenting, if this is the only data collection
> you're doing the change to Histograms.json should be sufficient
> documentation.
> 
> I'd really like you to be more specific about the user value: "measure the
> effectiveness and utility of the product for users" is still pretty vague.
> What specific product improvements will be driven by this data? Note that
> our own KPIs or understanding of users are not considered user value until
> we can tie them to more specific product improvements that will be driven by
> this data.

Currently we have no understanding of how the number of users who open the panel compares to the number of users who engage with the panel (that data we get from the loop server). 
Collecting this data will allow us to understand what that ratio looks like outside or marketing campaigns and during marketing campaigns in order to make the following decisions:
- Prioritization of panel improvements to increase the engagement with the panel once opened 
- Improvements to marketing messages which may currently end up getting with users opening the panel but doing nothing with it (snippets can get users to open the panel although we do not understand how the "open panel"/"engage with panel" ratio gets impacted during marketing campaigns)

> 
> You've said that Romain will be monitoring the dashboards... who will be
> building them and is there already a specification of what they will contain?
> 
We will need help for this and we don't currently have knowledge around how to extract Telemetry data. Do you have a suggestion regarding someone who could help us with it so we can scope data extraction and visualization of data for this and bug 1247268 which will require similar skill set.

> Also I'll note that you have this expiring in Firefox 50, which is
> relatively quickly. I expect you want this to be on for at least 6 months in
> release, which would be at least Firefox 51.

Thanks, indeed 6 months seems appropriate.
Flags: needinfo?(rtestard)
Rank: 24 → 15
Priority: P2 → P1
Comment on attachment 8718297 [details] [diff] [review]
MC_Patch

data-review=me

Ryan VanderMeulen's data engineering team is here to help everyone with analysis needs. Roberto or somebody on that team can help train/mentor on building analysis and reporting.
Attachment #8718297 - Flags: review?(benjamin) → feedback+
Comment on attachment 8720701 [details] [review]
[loop] mancas:bug1208416_2 > mozilla:master

Hey Ed, this patch is just to update the name of the histogram, could you review it?
Attachment #8720701 - Flags: review?(edilee)
Attached patch mc.patch (obsolete) — Splinter Review
Histogram name and expired date updated
Attachment #8720703 - Flags: feedback?(benjamin)
Attachment #8718297 - Attachment is obsolete: true
Comment on attachment 8720701 [details] [review]
[loop] mancas:bug1208416_2 > mozilla:master

Separate from this, should we change the alert_email address on the probe to not be mdeboer?
Attachment #8720701 - Flags: review?(edilee) → review+
Attachment #8720703 - Flags: feedback?(benjamin) → feedback+
Attached patch mc.patch (obsolete) — Splinter Review
n_values added
Attachment #8721239 - Flags: feedback+
Attachment #8720703 - Attachment is obsolete: true
Blocks: 1249646
No longer blocks: 1248602
https://hg.mozilla.org/integration/fx-team/rev/929ccd371714d1b403002b167fe6c416b3d8648b
Bug 1208416 - Report on Hello MAUs per new MAU definition. f=bsmedberg, r=mikedeboer
Attachment #8721239 - Attachment is obsolete: true
Attachment #8721979 - Flags: review?(mdeboer)
Attachment #8721979 - Flags: review?(mdeboer) → review+
Flags: needinfo?(b.mcb)
Whiteboard: [btpp-follow-up-2016-03-08]
Patch updated landed in master: https://github.com/mozilla/loop/commit/42f33dd37dc3607361dd8108fc630fd5eb71795c
Flags: needinfo?(b.mcb)
https://hg.mozilla.org/integration/fx-team/rev/398739346a3a872fd5aaf09098243b3dc8c9f4af
Bug 1208416 - Report on Hello MAUs per new MAU definition. r=mikedeboer,data-review=bsmedberg
Iteration: --- → 47.3 - Mar 7
Comment on attachment 8721979 [details] [diff] [review]
0001-Bug-1208416-Report-on-Hello-MAUs-per-new-MAU-definit.patch

Approval Request Comment
[Feature/regressing bug #]: Firefox Hello, metrics analysis
[User impact if declined]: Helps us to understand the Hello userbase, who is using it.
[Describe test coverage new/current, TreeHerder]: Landed in fx-team, telemetry has tests that check the results.
[Risks and why]: Low, simple addition of new metric for Hello.
[String/UUID change made/needed]: None

Note: the add-on parts of this will happen in a future merge.
Attachment #8721979 - Flags: approval-mozilla-aurora?
Comment on attachment 8721979 [details] [diff] [review]
0001-Bug-1208416-Report-on-Hello-MAUs-per-new-MAU-definit.patch

Adds some telemetry stuff for Hello, let's stuff it into last minute aurora
Attachment #8721979 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
https://hg.mozilla.org/mozilla-central/rev/398739346a3a
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Depends on: 1254958
Blocks: 1255012
You need to log in before you can comment on or make changes to this bug.