1148381 - (loop-metrics) Hello desktop client metrics

Status: RESOLVED INCOMPLETE

(Hello (Loop) :: Client, defect, P3)

Comment 1

[Romain Testard [:RT]]

Does this approach look right to tackle all dependent bugs at once?

Comment 2

[Jared Wein [:jaws] (please needinfo? me)]

Looks fine to me.

Do we know how we will interpret all of this data though? I'm not sure what UI/UX decisions we can make if we find out that long conversations have more face-muting than shorter conversations (in fact, the point that the conversation is longer will provide more ability for many of these types of actions).

In other words, even if we had all of this data collected, would we look at altering our UI to maximize some of these behaviors while trying to minimize others? Or are we looking at using this data to track a heartbeat of the product's state of health?

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

After looking at the various different data requested, it occurs to me that instead of having a metrics driver slaved to the room store (which we had been speculating was the way forward), that we'll want to instead have a metrics store itself with a bunch of new actions.  It's not yet clear to me whether the sdkDriver would still do some of the accounting and dispatch actions which would be posted by the metrics store, or if we'd want to move all of the metrics stuff out of the sdkDriver into the store. I'd be curious about thoughts on this, particularly from Standard8, and I think that might have some bearing on how we decide to batch these bugs.

Comment 4

[Jared Wein [:jaws] (please needinfo? me)]

Keeping all of the metrics tracking in one place would be really nice.

Comment 5

[Romain Testard [:RT]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #2)
> Looks fine to me.
> 
> Do we know how we will interpret all of this data though? I'm not sure what
> UI/UX decisions we can make if we find out that long conversations have more
> face-muting than shorter conversations (in fact, the point that the
> conversation is longer will provide more ability for many of these types of
> actions).
> 
> In other words, even if we had all of this data collected, would we look at
> altering our UI to maximize some of these behaviors while trying to minimize
> others? Or are we looking at using this data to track a heartbeat of the
> product's state of health?
I think it's both, this will allow us to understand scenarios such as:
- For some reason lots of people start face muting in very short conversations. This will probably identify that audio/video quality dropped to the point that people drop calls
- A large share of calls enabled video mute - will help identify that our default to start calls should be with video mute
- After a design change we identify that a lot less people use the mute feature, this will help us understand if the design changes impact the service in a positive manner

Comment 6

[:shell escalante]

will talk to dan about this and if the dependent bugs should be FHR rather than telemetry.

Comment 7

[Mark Banner (:standard8)]

(Commenting on User Story)
> This bug is about extending this capability to add more Hello related data
> (opt-out) that we'll be able to correlate in order to understand better how
> Hello gets used on the desktop client.

If this is all opt-out, then I propose that we get a block review and OK on it before we progress to implementing it.

A queries first though:

> per session duration category:
>   ** SHORTER_THAN_10S
>   ***Number of times the shared URL was accessed
>   ***Number of times the shared context was edited

Having these "per-call" means you won't find out if users have edited the context outside of a conversation, or if they accessed the url outside of it.

Comment 8

[Romain Testard [:RT]]

(In reply to Mark Banner (:standard8) from comment #7)
> (Commenting on User Story)
> > This bug is about extending this capability to add more Hello related data
> > (opt-out) that we'll be able to correlate in order to understand better how
> > Hello gets used on the desktop client.
> 
> If this is all opt-out, then I propose that we get a block review and OK on
> it before we progress to implementing it.
> 
> A queries first though:
> 
> > per session duration category:
> >   ** SHORTER_THAN_10S
> >   ***Number of times the shared URL was accessed
> >   ***Number of times the shared context was edited
> 
> Having these "per-call" means you won't find out if users have edited the
> context outside of a conversation, or if they accessed the url outside of it.

You're right, now changed the US to fix this, thanks!

Comment 9

[Dan Mosedale (:dmosedale, :dmose)]

(Commenting on User Story)
> As a UX designer or product manager, I want to understand how users use
> Firefox Hello so that I can make the right decisions.
> 
> This bug is aimed to bring several Hello Telemetry-related bugs together in
> order to have a common approach for them all. 
> Bug 1123660 helped adding Hello related data to FHR about the number
> successful sessions with a break-down per session duration bucket.

Standard8 and I spent some time talking about this, and decided that a good simple approach to start with would be to implement a telemetry store in desktop-only code.  If there ends up being sufficient benefit to generalizing it with the Google Analytics store (described in bug 1113222 comment 10) and factor out the transports into individual drivers, we can do that later.

As per discussion in today's meeting, after someone gets into this bug, it may make sense to spin off things that turn out to be complex, and it may also make sense to land one chunk of these in a first patch and spin off the rest in order to avoid this becoming too large.

>   ** SHORTER_THAN_10S
>   ***Number of "Face mute/unmute" events

As per today's call, if it's as easy or easier, there's probably a bit of benefit to framing this as "number of face mute/un-mute events per call", since that would allow deeper analysis, if that turns out to be necessary.  I assume, too, that this (as well as all the other descriptions in this user story with two actions separated by a /) is really two statistics: one about face-mutes and one about face unmutes.  Is that correct?

>   ***Number of "Audio mute/unmute" events
>   ***Number of "Window sharing initiated/stopped" events
>   ***Number of "Tab sharing initiated/stopped" events
>   ***Number of "Leave room" events
>   ***Number of "Close room" events

What's the difference between leave room and close room events?  Is the former about clicking the leave button, while the latter is about clicking the X button in the upper right corner of the window?

Comment 10

[Romain Testard [:RT]]

(In reply to Dan Mosedale (:dmose) - needinfo? me for response from comment #9)
> (Commenting on User Story)
> > As a UX designer or product manager, I want to understand how users use
> > Firefox Hello so that I can make the right decisions.
> > 
> > This bug is aimed to bring several Hello Telemetry-related bugs together in
> > order to have a common approach for them all. 
> > Bug 1123660 helped adding Hello related data to FHR about the number
> > successful sessions with a break-down per session duration bucket.
> 
> Standard8 and I spent some time talking about this, and decided that a good
> simple approach to start with would be to implement a telemetry store in
> desktop-only code.  If there ends up being sufficient benefit to
> generalizing it with the Google Analytics store (described in bug 1113222
> comment 10) and factor out the transports into individual drivers, we can do
> that later.
> 
> As per discussion in today's meeting, after someone gets into this bug, it
> may make sense to spin off things that turn out to be complex, and it may
> also make sense to land one chunk of these in a first patch and spin off the
> rest in order to avoid this becoming too large.
> 
> >   ** SHORTER_THAN_10S
> >   ***Number of "Face mute/unmute" events
> 
> As per today's call, if it's as easy or easier, there's probably a bit of
> benefit to framing this as "number of face mute/un-mute events per call",
> since that would allow deeper analysis, if that turns out to be necessary. 
> I assume, too, that this (as well as all the other descriptions in this user
> story with two actions separated by a /) is really two statistics: one about
> face-mutes and one about face unmutes.  Is that correct?
> 
Yes, added clarification in the US so it's obvious.

> >   ***Number of "Audio mute/unmute" events
> >   ***Number of "Window sharing initiated/stopped" events
> >   ***Number of "Tab sharing initiated/stopped" events
> >   ***Number of "Leave room" events
> >   ***Number of "Close room" events
> 
> What's the difference between leave room and close room events?  Is the
> former about clicking the leave button, while the latter is about clicking
> the X button in the upper right corner of the window?

Yes, added clarification in the US so it's obvious.

Comment 11

[Romain Testard [:RT]]

FYI Sam create a good doc helping understand how to get the reporting data from FHR: https://mana.mozilla.org/wiki/display/CLOUDSERVICES/Getting+set+up+to+run+your+first+FHR+analysis

Comment 12

[Mark Banner (:standard8)]

When it comes to implementation:

Bug 1132222 implemented a standaloneMetricsStore, we might want to consider moving that to a driver and have the work here also implemented as a driver, the drivers would have the same API and be passed to the activeRoomStore.

I'm not 100% sure this will make the most sense. The parts to really look at are if it would make the _checkRoomState and _checkMuteState functions easier, and help the long-term maintenance of the code when we're updating/extending room states.

If it doesn't make sense, then probably leaving as separate stores is reasonable for now.

Comment 13

[:shell escalante]

RT to attach narrower scope flow visual and limit to Quality Metrics (break out other metrics in separate bug).  Mark looking at feasibility of the metrics in the visual diagram (to be attached)

Comment 14

[:shell escalante]

moving lower - but we are taking bits and pieces as we go rather than whole bug.  keeping for meta.

Comment 15

["Saptarshi Guha[:joy]"]

Hello,
is any of this data in the FHR already? Will it be in Nightly only or will it be on for Release profiles too?
Thanks
Saptarshi

Comment 16

[Romain Testard [:RT]]

We have the following:
Bug 1127574 (FF41):
- "Create room" events
- "Delete room" events
- "Copy room URL from panel" events
- "Copy room URL from conversation window" events
- "E-mail room URL from conversation window" events
- "E-mail room URL from error panel after a direct call failed" events 

Bug 1142520 (FF41):
Telemetry events should be sent when:
- Conversations are created with context
- Context URLs are accessed from the conversation window
- Context is added from within the conversation window

Bug 1135095 (FF39):
Send a Telemetry event in the following scenarios:
- Tab sharing enabled
- Tab sharing disabled
- Window sharing enabled
- Window sharing disabled      

We want these in GA since user behavior is different between GA and Nightly.

Comment 17

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.