Firefox Hello Metrics Needed for Growth Model

RESOLVED INCOMPLETE

Status

P3
normal
RESOLVED INCOMPLETE
4 years ago
2 years ago

People

(Reporter: garethc, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [fxgrowth])

(Reporter)

Description

4 years ago
Hi,

The Growth Team is currently building a Firefox Hello growth model and we are in need of some metrics to understand where we stand from a baseline and how we can influence those measures over time.

Stats Needed Include:
1) # of Firefox Hello Active Users
2) # of Firefox Users Opening Firefox Hello (defined as clicking 'start a conversation' or a conversation link from within the drop down menu)
3) # of Users Importing their Address Book
4) # of Users Clicking Email / Copy Link (Combined)
5) # of Total Connected Conversations 


Data Dimensions
Can we get the above data split out on a weekly basis

Data Format:
Can we get this either as a csv or json file?

Report Frequency:
Can we have this data refreshed weekly?

Any questions, please let me know. Thanks.

Gareth

Updated

4 years ago
Component: Metrics: Dashboard → Metrics: Pipeline
Assignee: nobody → kparlante
Priority: -- → P2

Comment 1

4 years ago
Hi Gareth, a few questions and comments:

1) What do you mean by "Hello Active Users"? The definition in the metrics team's data dictionary is "A profile that has one or more call entries in FHR". That data hasn't landed yet, it was implemented in https://bugzilla.mozilla.org/show_bug.cgi?id=1123660. We're also working on a server based definition, which counts the number of unique users who have successful conversations (rooms or calls). The variant of the server based metric that we have right now is "number of unique users who are in a room at the same time as another room, or who are on a call that reports successful media setup on both sides". (There is a weekly json available for that one, combined 7 day active: https://metrics.services.mozilla.com/loop-server-dashboard/)

2) "Opening Firefox Hello" -- again, need to make a call on whether or not you want a FHR based metric or a server side metric. FWIW, The https://metrics.services.mozilla.com/loop-server-dashboard/ has a room funnel; we could do a similar filter that counted unique users at various stages instead of unique rooms.

3, 4) This needs to come from FHR data. Romain will have a better sense of what is landing when, here's a meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1148381

5) As in #1, need to make a decision on whether to use FHR based data (https://bugzilla.mozilla.org/show_bug.cgi?id=1123660) or server side data.

It might be worth setting up a conversation with Romain and Saptarshi to make sure the model is consistent with executive/product dashboard decisions.
(Reporter)

Comment 2

4 years ago
Thanks Katie. My responses are inline below.

> 1) What do you mean by "Hello Active Users"? The definition in the metrics
> team's data dictionary is "A profile that has one or more call entries in
> FHR". That data hasn't landed yet, it was implemented in
> https://bugzilla.mozilla.org/show_bug.cgi?id=1123660. 

Let's use the team's data dictionary definition for consistency. When do you expect the data to land?
 
> 3, 4) This needs to come from FHR data. Romain will have a better sense of
> what is landing when, here's a meta bug:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1148381

I'd like to use one consistent source across all of these metrics.  So, if data only comes from FHR as in #3 & #4, then I guess we should probably go with that route. 

 
> 5) As in #1, need to make a decision on whether to use FHR based data
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1123660) or server side data.

Let's use FHR

> It might be worth setting up a conversation with Romain and Saptarshi to
> make sure the model is consistent with executive/product dashboard decisions.

I will set that up.

Do you have a rough ballpark estimate as to when this data will be fully available?

Comment 3

4 years ago
Looks like https://bugzilla.mozilla.org/show_bug.cgi?id=1123660 will ride the trains with 39. Romain might know about #2/3/4
Depends on: 1123660
Flags: needinfo?(rtestard)

Updated

4 years ago
Whiteboard: [fxgrowth]
(In reply to Gareth Cull [:garethc] from comment #2)
> Thanks Katie. My responses are inline below.
> 
> > 1) What do you mean by "Hello Active Users"? The definition in the metrics
> > team's data dictionary is "A profile that has one or more call entries in
> > FHR". That data hasn't landed yet, it was implemented in
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1123660. 
> 
> Let's use the team's data dictionary definition for consistency. When do you
> expect the data to land?
This is planned for Fx39 GA.
Our definition of an active user is a user who has had a successful call or room session with someone else in the last 30 days. This is currently collected server side through https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_combined_monthly_unique_callers.json  (visible on https://metrics.services.mozilla.com/accounts-exec-dashboard/ under "Hello MAU").
The FHR data should give us that too.
>  
> > 3, 4) This needs to come from FHR data. Romain will have a better sense of
> > what is landing when, here's a meta bug:
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1148381
> 
> I'd like to use one consistent source across all of these metrics.  So, if
> data only comes from FHR as in #3 & #4, then I guess we should probably go
> with that route. 
> 
I have bug 1148381 opened which will address 2/3/4 and others. I have discussed with Benjamin and need to answer a bunch of questions he had to clear privacy concerns.
I'm trying to get this in for Fx40.
>  
> > 5) As in #1, need to make a decision on whether to use FHR based data
> > (https://bugzilla.mozilla.org/show_bug.cgi?id=1123660) or server side data.
> 
> Let's use FHR
Agreed. In the mean time we have https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_room_sessions.json
> 
> > It might be worth setting up a conversation with Romain and Saptarshi to
> > make sure the model is consistent with executive/product dashboard decisions.
> 
> I will set that up.
> 
> Do you have a rough ballpark estimate as to when this data will be fully
> available?
Flags: needinfo?(rtestard)

Comment 5

4 years ago
Waiting on the data to be available. We should probably create individual bugs for each filter once the data is ready.
Assignee: kparlante → nobody
Depends on: 1148381
Depends on: 1132222

Updated

4 years ago
Assignee: nobody → spenrose

Updated

4 years ago
Assignee: spenrose → nobody
Priority: P2 → P3

Updated

4 years ago
Component: Metrics: Pipeline → Metrics: Product Metrics
Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.