1146616 - Firefox Hello Metrics Needed for Growth Model

Status: RESOLVED INCOMPLETE

(Cloud Services :: Metrics: Product Metrics, defect, P3)

Description

[Gareth Cull [:garethc]]

Hi,

The Growth Team is currently building a Firefox Hello growth model and we are in need of some metrics to understand where we stand from a baseline and how we can influence those measures over time.

Stats Needed Include:
1) # of Firefox Hello Active Users
2) # of Firefox Users Opening Firefox Hello (defined as clicking 'start a conversation' or a conversation link from within the drop down menu)
3) # of Users Importing their Address Book
4) # of Users Clicking Email / Copy Link (Combined)
5) # of Total Connected Conversations 


Data Dimensions
Can we get the above data split out on a weekly basis

Data Format:
Can we get this either as a csv or json file?

Report Frequency:
Can we have this data refreshed weekly?

Any questions, please let me know. Thanks.

Gareth

Comment 1

[Katie Parlante]

Hi Gareth, a few questions and comments:

1) What do you mean by "Hello Active Users"? The definition in the metrics team's data dictionary is "A profile that has one or more call entries in FHR". That data hasn't landed yet, it was implemented in https://bugzilla.mozilla.org/show_bug.cgi?id=1123660. We're also working on a server based definition, which counts the number of unique users who have successful conversations (rooms or calls). The variant of the server based metric that we have right now is "number of unique users who are in a room at the same time as another room, or who are on a call that reports successful media setup on both sides". (There is a weekly json available for that one, combined 7 day active: https://metrics.services.mozilla.com/loop-server-dashboard/)

2) "Opening Firefox Hello" -- again, need to make a call on whether or not you want a FHR based metric or a server side metric. FWIW, The https://metrics.services.mozilla.com/loop-server-dashboard/ has a room funnel; we could do a similar filter that counted unique users at various stages instead of unique rooms.

3, 4) This needs to come from FHR data. Romain will have a better sense of what is landing when, here's a meta bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1148381

5) As in #1, need to make a decision on whether to use FHR based data (https://bugzilla.mozilla.org/show_bug.cgi?id=1123660) or server side data.

It might be worth setting up a conversation with Romain and Saptarshi to make sure the model is consistent with executive/product dashboard decisions.

Comment 2

[Gareth Cull [:garethc]]

Thanks Katie. My responses are inline below.

> 1) What do you mean by "Hello Active Users"? The definition in the metrics
> team's data dictionary is "A profile that has one or more call entries in
> FHR". That data hasn't landed yet, it was implemented in
> https://bugzilla.mozilla.org/show_bug.cgi?id=1123660. 

Let's use the team's data dictionary definition for consistency. When do you expect the data to land?
 
> 3, 4) This needs to come from FHR data. Romain will have a better sense of
> what is landing when, here's a meta bug:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1148381

I'd like to use one consistent source across all of these metrics.  So, if data only comes from FHR as in #3 & #4, then I guess we should probably go with that route. 

 
> 5) As in #1, need to make a decision on whether to use FHR based data
> (https://bugzilla.mozilla.org/show_bug.cgi?id=1123660) or server side data.

Let's use FHR

> It might be worth setting up a conversation with Romain and Saptarshi to
> make sure the model is consistent with executive/product dashboard decisions.

I will set that up.

Do you have a rough ballpark estimate as to when this data will be fully available?

Comment 3

[Katie Parlante]

Looks like https://bugzilla.mozilla.org/show_bug.cgi?id=1123660 will ride the trains with 39. Romain might know about #2/3/4

Comment 4

[Romain Testard [:RT]]

(In reply to Gareth Cull [:garethc] from comment #2)
> Thanks Katie. My responses are inline below.
> 
> > 1) What do you mean by "Hello Active Users"? The definition in the metrics
> > team's data dictionary is "A profile that has one or more call entries in
> > FHR". That data hasn't landed yet, it was implemented in
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1123660. 
> 
> Let's use the team's data dictionary definition for consistency. When do you
> expect the data to land?
This is planned for Fx39 GA.
Our definition of an active user is a user who has had a successful call or room session with someone else in the last 30 days. This is currently collected server side through https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_combined_monthly_unique_callers.json  (visible on https://metrics.services.mozilla.com/accounts-exec-dashboard/ under "Hello MAU").
The FHR data should give us that too.
>  
> > 3, 4) This needs to come from FHR data. Romain will have a better sense of
> > what is landing when, here's a meta bug:
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1148381
> 
> I'd like to use one consistent source across all of these metrics.  So, if
> data only comes from FHR as in #3 & #4, then I guess we should probably go
> with that route. 
> 
I have bug 1148381 opened which will address 2/3/4 and others. I have discussed with Benjamin and need to answer a bunch of questions he had to clear privacy concerns.
I'm trying to get this in for Fx40.
>  
> > 5) As in #1, need to make a decision on whether to use FHR based data
> > (https://bugzilla.mozilla.org/show_bug.cgi?id=1123660) or server side data.
> 
> Let's use FHR
Agreed. In the mean time we have https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_room_sessions.json
> 
> > It might be worth setting up a conversation with Romain and Saptarshi to
> > make sure the model is consistent with executive/product dashboard decisions.
> 
> I will set that up.
> 
> Do you have a rough ballpark estimate as to when this data will be fully
> available?

Comment 5

[Katie Parlante]

Waiting on the data to be available. We should probably create individual bugs for each filter once the data is ready.

Comment 6

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.