Bug 1148381 (loop-metrics)

Hello desktop client metrics

RESOLVED INCOMPLETE

Status

Hello (Loop)
Client
P3
normal
Rank:
35
RESOLVED INCOMPLETE
3 years ago
a year ago

People

(Reporter: RT, Unassigned)

Tracking

unspecified
Points:
---
Dependency tree / graph
Bug Flags:
firefox-backlog +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [metrics])

User Story

As a UX designer or product manager, I want to understand how users use Firefox Hello so that I can make the right decisions.

This bug is aimed to bring several Hello Telemetry-related bugs together in order to have a common approach for them all. 
Bug 1123660 helped adding Hello related data to FHR about the number successful sessions with a break-down per session duration bucket.

This bug is about extending this capability to add more Hello related data (opt-out) that we'll be able to correlate in order to understand better how Hello gets used on the desktop client.

The proposal is to have the following Hello related data added to FHR:
-The following data gets added to FHR
  * Number of Hello sessions where media connection completed on both sides, per session duration category:
  ** SHORTER_THAN_10S
  ***Number of "Face mute" events
  ***Number of "Face unmute" events
  ***Number of "Audio mute" events
  ***Number of "Audio unmute" events
  ***Number of "Window sharing initiated" events
  ***Number of "Window sharing stopped" events
  ***Number of "Tab sharing initiated" events
  ***Number of "Tab sharing stopped" events
  ***Number of "Leave room" events (leave button)
  ***Number of "Close room" events (X button in the upper right corner of the window)
  ** 10S_TO_30S
  ***Number of "Face mute" events
  ***Number of "Face unmute" events
  ***Number of "Audio mute" events
  ***Number of "Audio unmute" events
  ***Number of "Window sharing initiated" events
  ***Number of "Window sharing stopped" events
  ***Number of "Tab sharing initiated" events
  ***Number of "Tab sharing stopped" events
  ***Number of "Leave room" events (leave button)
  ***Number of "Close room" events (X button in the upper right corner of the window)
  ** 30S_TO_5M
  ***Number of "Face mute" events
  ***Number of "Face unmute" events
  ***Number of "Audio mute" events
  ***Number of "Audio unmute" events
  ***Number of "Window sharing initiated" events
  ***Number of "Window sharing stopped" events
  ***Number of "Tab sharing initiated" events
  ***Number of "Tab sharing stopped" events
  ***Number of "Leave room" events (leave button)
  ***Number of "Close room" events (X button in the upper right corner of the window)
  ** 5M_OR_LONGER     
  ***Number of "Face mute" events
  ***Number of "Face unmute" events
  ***Number of "Audio mute" events
  ***Number of "Audio unmute" events
  ***Number of "Window sharing initiated" events
  ***Number of "Window sharing stopped" events
  ***Number of "Tab sharing initiated" events
  ***Number of "Tab sharing stopped" events
  ***Number of "Leave room" events (leave button)
  ***Number of "Close room" events (X button in the upper right corner of the window)
  * Other
  **Number of "Create room without context" events
  **Number of "Create room with context" events
  **Number of "access context URL" was events
  **Number of "edit context URL" events
  **Number of "Delete room" events
  **Number of "Copy room URL from panel" events
  **Number of "Copy room URL from conversation window" events
  **Number of "E-mail room URL from conversation window" events
  **Number of "E-mail room URL from error panel after a direct call failed" events   
  **Number of "Rename room from panel" events
  **Number of "Rename room from conversation window" events
  **Number of "Change availability status" events
  **Number of "Import contacts" events
  **Number of "New contact manually created" events
  **Number of "Contact deleted" events
  **Number of "Contact blocked" events
  **Number of "Contact amended" events
  **Number of "Share Hello URL through Shareplane" events
  **Number of "Use help button from gear menu" events
  **Number of "Use Tour button from gear menu" events
  **Number of "Access my FxA account page from the gear menu" events
  **Number of "Use Sign-in button from gear menu" events
  **Number of "Use Sign-in button from panel" events
  **Number of "Open panel" events
  **Number of "Open conversation" events
Comment hidden (empty)
(Reporter)

Updated

3 years ago
(Reporter)

Updated

3 years ago
User Story: (updated)
(Reporter)

Updated

3 years ago
User Story: (updated)
(Reporter)

Comment 1

3 years ago
Does this approach look right to tackle all dependent bugs at once?
Flags: needinfo?(standard8)
Flags: needinfo?(jaws)
Flags: needinfo?(dmose)
Looks fine to me.

Do we know how we will interpret all of this data though? I'm not sure what UI/UX decisions we can make if we find out that long conversations have more face-muting than shorter conversations (in fact, the point that the conversation is longer will provide more ability for many of these types of actions).

In other words, even if we had all of this data collected, would we look at altering our UI to maximize some of these behaviors while trying to minimize others? Or are we looking at using this data to track a heartbeat of the product's state of health?
Flags: needinfo?(jaws) → needinfo?(rtestard)
After looking at the various different data requested, it occurs to me that instead of having a metrics driver slaved to the room store (which we had been speculating was the way forward), that we'll want to instead have a metrics store itself with a bunch of new actions.  It's not yet clear to me whether the sdkDriver would still do some of the accounting and dispatch actions which would be posted by the metrics store, or if we'd want to move all of the metrics stuff out of the sdkDriver into the store. I'd be curious about thoughts on this, particularly from Standard8, and I think that might have some bearing on how we decide to batch these bugs.
Flags: needinfo?(dmose) → needinfo?(jaws)
Keeping all of the metrics tracking in one place would be really nice.
Flags: needinfo?(jaws)
(Reporter)

Updated

3 years ago
Flags: needinfo?(rtestard)
(Reporter)

Updated

3 years ago
User Story: (updated)
(Reporter)

Comment 5

3 years ago
(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #2)
> Looks fine to me.
> 
> Do we know how we will interpret all of this data though? I'm not sure what
> UI/UX decisions we can make if we find out that long conversations have more
> face-muting than shorter conversations (in fact, the point that the
> conversation is longer will provide more ability for many of these types of
> actions).
> 
> In other words, even if we had all of this data collected, would we look at
> altering our UI to maximize some of these behaviors while trying to minimize
> others? Or are we looking at using this data to track a heartbeat of the
> product's state of health?
I think it's both, this will allow us to understand scenarios such as:
- For some reason lots of people start face muting in very short conversations. This will probably identify that audio/video quality dropped to the point that people drop calls
- A large share of calls enabled video mute - will help identify that our default to start calls should be with video mute
- After a design change we identify that a lot less people use the mute feature, this will help us understand if the design changes impact the service in a positive manner

Comment 6

3 years ago
will talk to dan about this and if the dependent bugs should be FHR rather than telemetry.
Rank: 19
Flags: firefox-backlog+
Priority: -- → P1
(Commenting on User Story)
> This bug is about extending this capability to add more Hello related data
> (opt-out) that we'll be able to correlate in order to understand better how
> Hello gets used on the desktop client.

If this is all opt-out, then I propose that we get a block review and OK on it before we progress to implementing it.

A queries first though:

> per session duration category:
>   ** SHORTER_THAN_10S
>   ***Number of times the shared URL was accessed
>   ***Number of times the shared context was edited

Having these "per-call" means you won't find out if users have edited the context outside of a conversation, or if they accessed the url outside of it.
Flags: needinfo?(standard8)
(Reporter)

Comment 8

3 years ago
(In reply to Mark Banner (:standard8) from comment #7)
> (Commenting on User Story)
> > This bug is about extending this capability to add more Hello related data
> > (opt-out) that we'll be able to correlate in order to understand better how
> > Hello gets used on the desktop client.
> 
> If this is all opt-out, then I propose that we get a block review and OK on
> it before we progress to implementing it.
> 
> A queries first though:
> 
> > per session duration category:
> >   ** SHORTER_THAN_10S
> >   ***Number of times the shared URL was accessed
> >   ***Number of times the shared context was edited
> 
> Having these "per-call" means you won't find out if users have edited the
> context outside of a conversation, or if they accessed the url outside of it.

You're right, now changed the US to fix this, thanks!
User Story: (updated)
(Reporter)

Updated

3 years ago
User Story: (updated)

Updated

3 years ago
Whiteboard: [loop-metrics]
Alias: loop-metrics
(Commenting on User Story)
> As a UX designer or product manager, I want to understand how users use
> Firefox Hello so that I can make the right decisions.
> 
> This bug is aimed to bring several Hello Telemetry-related bugs together in
> order to have a common approach for them all. 
> Bug 1123660 helped adding Hello related data to FHR about the number
> successful sessions with a break-down per session duration bucket.

Standard8 and I spent some time talking about this, and decided that a good simple approach to start with would be to implement a telemetry store in desktop-only code.  If there ends up being sufficient benefit to generalizing it with the Google Analytics store (described in bug 1113222 comment 10) and factor out the transports into individual drivers, we can do that later.

As per discussion in today's meeting, after someone gets into this bug, it may make sense to spin off things that turn out to be complex, and it may also make sense to land one chunk of these in a first patch and spin off the rest in order to avoid this becoming too large.

>   ** SHORTER_THAN_10S
>   ***Number of "Face mute/unmute" events

As per today's call, if it's as easy or easier, there's probably a bit of benefit to framing this as "number of face mute/un-mute events per call", since that would allow deeper analysis, if that turns out to be necessary.  I assume, too, that this (as well as all the other descriptions in this user story with two actions separated by a /) is really two statistics: one about face-mutes and one about face unmutes.  Is that correct?

>   ***Number of "Audio mute/unmute" events
>   ***Number of "Window sharing initiated/stopped" events
>   ***Number of "Tab sharing initiated/stopped" events
>   ***Number of "Leave room" events
>   ***Number of "Close room" events

What's the difference between leave room and close room events?  Is the former about clicking the leave button, while the latter is about clicking the X button in the upper right corner of the window?
Flags: needinfo?(rtestard)
(Reporter)

Comment 10

3 years ago
(In reply to Dan Mosedale (:dmose) - needinfo? me for response from comment #9)
> (Commenting on User Story)
> > As a UX designer or product manager, I want to understand how users use
> > Firefox Hello so that I can make the right decisions.
> > 
> > This bug is aimed to bring several Hello Telemetry-related bugs together in
> > order to have a common approach for them all. 
> > Bug 1123660 helped adding Hello related data to FHR about the number
> > successful sessions with a break-down per session duration bucket.
> 
> Standard8 and I spent some time talking about this, and decided that a good
> simple approach to start with would be to implement a telemetry store in
> desktop-only code.  If there ends up being sufficient benefit to
> generalizing it with the Google Analytics store (described in bug 1113222
> comment 10) and factor out the transports into individual drivers, we can do
> that later.
> 
> As per discussion in today's meeting, after someone gets into this bug, it
> may make sense to spin off things that turn out to be complex, and it may
> also make sense to land one chunk of these in a first patch and spin off the
> rest in order to avoid this becoming too large.
> 
> >   ** SHORTER_THAN_10S
> >   ***Number of "Face mute/unmute" events
> 
> As per today's call, if it's as easy or easier, there's probably a bit of
> benefit to framing this as "number of face mute/un-mute events per call",
> since that would allow deeper analysis, if that turns out to be necessary. 
> I assume, too, that this (as well as all the other descriptions in this user
> story with two actions separated by a /) is really two statistics: one about
> face-mutes and one about face unmutes.  Is that correct?
> 
Yes, added clarification in the US so it's obvious.

> >   ***Number of "Audio mute/unmute" events
> >   ***Number of "Window sharing initiated/stopped" events
> >   ***Number of "Tab sharing initiated/stopped" events
> >   ***Number of "Leave room" events
> >   ***Number of "Close room" events
> 
> What's the difference between leave room and close room events?  Is the
> former about clicking the leave button, while the latter is about clicking
> the X button in the upper right corner of the window?

Yes, added clarification in the US so it's obvious.
User Story: (updated)
Flags: needinfo?(rtestard)

Updated

3 years ago
Blocks: 1146616
(Reporter)

Comment 11

3 years ago
FYI Sam create a good doc helping understand how to get the reporting data from FHR: https://mana.mozilla.org/wiki/display/CLOUDSERVICES/Getting+set+up+to+run+your+first+FHR+analysis
When it comes to implementation:

Bug 1132222 implemented a standaloneMetricsStore, we might want to consider moving that to a driver and have the work here also implemented as a driver, the drivers would have the same API and be passed to the activeRoomStore.

I'm not 100% sure this will make the most sense. The parts to really look at are if it would make the _checkRoomState and _checkMuteState functions easier, and help the long-term maintenance of the code when we're updating/extending room states.

If it doesn't make sense, then probably leaving as separate stores is reasonable for now.
Whiteboard: [loop-metrics] → [loop-metrics][implementing? Ready comment 12]

Updated

3 years ago
Rank: 19 → 27
Priority: P1 → P2

Comment 13

3 years ago
RT to attach narrower scope flow visual and limit to Quality Metrics (break out other metrics in separate bug).  Mark looking at feasibility of the metrics in the visual diagram (to be attached)
Rank: 27 → 19
Flags: needinfo?(rtestard)
Priority: P2 → P1
Whiteboard: [loop-metrics][implementing? Ready comment 12] → [metrics]

Updated

3 years ago
Rank: 19 → 26
Priority: P1 → P2

Comment 14

3 years ago
moving lower - but we are taking bits and pieces as we go rather than whole bug.  keeping for meta.
Rank: 26 → 35
Priority: P2 → P3
Hello,
is any of this data in the FHR already? Will it be in Nightly only or will it be on for Release profiles too?
Thanks
Saptarshi
(Reporter)

Comment 16

3 years ago
We have the following:
Bug 1127574 (FF41):
- "Create room" events
- "Delete room" events
- "Copy room URL from panel" events
- "Copy room URL from conversation window" events
- "E-mail room URL from conversation window" events
- "E-mail room URL from error panel after a direct call failed" events 

Bug 1142520 (FF41):
Telemetry events should be sent when:
- Conversations are created with context
- Context URLs are accessed from the conversation window
- Context is added from within the conversation window

Bug 1135095 (FF39):
Send a Telemetry event in the following scenarios:
- Tab sharing enabled
- Tab sharing disabled
- Window sharing enabled
- Window sharing disabled      

We want these in GA since user behavior is different between GA and Nightly.
Flags: needinfo?(rtestard)
Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.