1041439 - Collection of call metadata along with user feedback form

Status: RESOLVED WONTFIX

(Hello (Loop) :: Client, defect, P2)

Description

[Romain Testard [:RT]]

The feedback form should expose a checkbox allowing the user to decide whether he agrees to send a set of call metadata with his feedback as part of the feedback form.

The proposed set of data collected is:
    * Client type (Desktop client, Link clicker UI, FFOS)
    * OS
    * Calling mode when call terminates (send audio receive audio, send audio receive video, send video receive audio, send video receive video)
    * Audio and Video Codec
    * Call duration (call duration when looked as an aggregate is a good reflection of call quality - the shorter the average call length the worse the call quality is)
    * Call disconnection cause (error occurred with error code, remote peer terminated, local peer terminated)

Comment 1

[Romain Testard [:RT]]

Adam, can you please comment on feasibility for collection of the following:
* Are averaged A/V quality metrics useful and reflective of the call experience
* Is-it possible today to obtain data about the remote peer's client type and remote peers' OS?

Comment 2

[Adam Roach [:abr]]

(In reply to Romain Testard [:RT] from comment #1)
> Adam, can you please comment on feasibility for collection of the following:
> * Are averaged A/V quality metrics useful and reflective of the call
> experience

For most call defects, you're going to have reasonable metrics for most of the call, combined with bursts of really bad quality. For such cases, averaged metrics generally don't tell us very much. What you would really want is something like time-series data sampled over the course of a call; or, some kind of min/max/avg from such sampling. But that's not hard.

There's a complication here. The way the SDK works, we can't get to the peer connections directly, which means we can't get to the call quality metrics. For the desktop client, we have some hacks in place that let us detect failures and grab corresponding ICE logs. We could do the same kind of thing for call quality, but only for the desktop client. We're left without any way to collect call metrics for the standalone client, unless TokBox wants to add an API for us to get at this information. So far, they've been reluctant to give that kind of direct access.

> * Is-it possible today to obtain data about the remote peer's client type
> and remote peers' OS?

Not today. Of course, we could add this kind of information to the Loop server call signaling. There may be some privacy considerations here; but I will note that, e.g., Thunderbird includes this information in the messages it sends.

Comment 3

[[:Cww]]

Can we get someone on the Privacy team to OK these? We store PII on input so we have to clear out data collection with them.

Since these aren't identifying, they're probably OK, but I don't get to make that call.

Comment 4

[Romain Testard [:RT]]

Thanks.
So the proposal is to collect the following when feedback is submitted through the feedback form and the user does not untick the check-box (unticking it means this data does not get collected):
    * Client type (Desktop client, Link clicker UI, FFOS)
    * OS type and version
    * Calling mode when call terminates (send audio receive audio, send audio receive video, send video receive audio, send video receive video)
    * Audio and Video Codec
    * Call duration (call duration when looked as an aggregate is a good reflection of call quality - the shorter the average call length the worse the call quality is)
    * Call disconnection cause (error occurred with error code, remote peer terminated, local peer terminated)
    * min/max/avg QoS metrics made from time-series data sampling over the course of the call [only applicable to desktop client users]

Mika, I needinfo you - could you please review and comment whether it is acceptable or not to collect the above data when the user sumits manual feedback given he has the option to opt-out via a tick box ?
UX will be the following with the addition of a checkbox underneath the sad and happy faces. The checkbox would be enabled by default and could say for instance "I agree to send fully anonymous data about my call to help improve the Firefox Hello product"
https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/#feedback

Comment 5

[Mika]

(In reply to Romain Testard [:RT] from comment #4)

The data collection listed in Comment 4 is fine.  For the UX:
- it's okay to have the box default ticked
- the text should say "Send Mozilla performance data to improve Hello.  Learn More"
- Learn More" should link to the Privacy Notice.  I'll make sure this is covered there.

Do you want users to see this permission request each time? It's fine to give the user the ability to "always send to Mozilla", ""never send to Mozilla", or "Ask each time"

Comment 6

[Mika]

(In reply to [:Cww] from comment #3)
> Can we get someone on the Privacy team to OK these? We store PII on input so
> we have to clear out data collection with them.
> 
> Since these aren't identifying, they're probably OK, but I don't get to make
> that call.

You are correct that the data collection described here, by itself, isn't PII.  What are we doing to ensure that it can't be combined with other data in other parts of Mozilla (for example, FxA data, IP addresses or location data) to identify the device or account?

Comment 7

[Romain Testard [:RT]]

(In reply to Mika from comment #5)
> (In reply to Romain Testard [:RT] from comment #4)
> 
> The data collection listed in Comment 4 is fine.  For the UX:
> - it's okay to have the box default ticked
> - the text should say "Send Mozilla performance data to improve Hello. 
> Learn More"
> - Learn More" should link to the Privacy Notice.  I'll make sure this is
> covered there.
> 
> Do you want users to see this permission request each time? It's fine to
> give the user the ability to "always send to Mozilla", ""never send to
> Mozilla", or "Ask each time"

I'll let Darrin make a call on the UX side. If we can prompt the user only once then great. Having 3 options may complexity a UI which already allows users to provide feedback and re-dial a disconnected call though.

Comment 8

[:shell escalante]

we'll see if we get enough benefit from bug 1024568 - and also the timing is better to gather info in the call.

Comment 9

[Matt Grimes [:Matt_G]]

Do we still need to capture additional data in Input or are we going to defer it as suggested in comment 8? Willkg needs to know soon since capturing that additional data requires work on the Input side.

Comment 10

[Romain Testard [:RT]]

(In reply to Matt Grimes [:Matt_G] from comment #9)
> Do we still need to capture additional data in Input or are we going to
> defer it as suggested in comment 8? Willkg needs to know soon since
> capturing that additional data requires work on the Input side.

Not initially, we may need this as part of our FF35 release but it seems we may rather want to implement a more involved/complete solution with bug 1024568. 
I think we need to review this with Adam who I needInfo here so we can make that decision which may require us to use another place to store the data as I understand Adam's bug requires a lot more than what is on this current bug.

Comment 11

[Romain Testard [:RT]]

Now with NI

Comment 12

["Saptarshi Guha[:joy]"]

If we can send usefuland actionable  call metrics with user feedback and remain privacy respectful
theen we should do it here.

Otherwise we'll be in a situation with user feedback and little idea of the actual data behind the call that caused such feedback.

Comment 13

[Romain Testard [:RT]]

Confirmed now after a call with Adam - We won't do this for our initial release and we'll look into the best solution for storing that data as well as data related to [1] as part of our FF35 release.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1024568

Comment 14

[[:Cww]]

Just confirming: we are still collecting feedback via Input though, just not with much metadata attached.

Comment 15

[Maire Reavy [:mreavy]]

How much of this data reporting is critical to have in Fx35?  (Are you getting some of this from sources other than the clients?)

Comment 16

[Romain Testard [:RT]]

I think bug 1024568 is higher priority than this as it is more automated is likely more useful in providing in providing data to fix issues.
I still think there is value in having data attached to the feedback form to help validate/categorize the feedback although not a blocker for Fx35.

Comment 17

[:shell escalante]

pushing to 37 - the other bug RT referenced is in 36 now.

Comment 18

[:shell escalante]

how much of this are we getting through other means?  Is this bug worth keeping?  didn't the telemetry folks say they weren't set up to store this type of data - or am i remembering a different conversation?

Adam broke down bug 1024568  - which sounds like a lot of work that will take several iterations (because of other priorities).  working on the components in Fx38 - but doubt it will be completed unless highly prioritized (asking for UX, seeing what platform work is needed, there's server component, there are front end client components).

Comment 19

[Adam Roach [:abr]]

(In reply to sescalante from comment #18)
> how much of this are we getting through other means?  Is this bug worth
> keeping?  didn't the telemetry folks say they weren't set up to store this
> type of data - or am i remembering a different conversation?

It gets complicated very quickly. The feedback forms go into fjord (https://input.mozilla.org/), which is a very different thing than telemetry. In conversations with Will and the rest of the fjord team, it's been pretty clear that they have serious misgivings about putting anything pertaining to technical data in this system, or adding anything that would allow correlation between user feedback and technical information. I'm ni'ing Will here to weigh in on the topic (with particular attention paid to in input from legal [comment 5] metrics [comment 12]).

Regardless of whether this is something we can get every to agree is okay, my preference (if this work remains high priority) would be to prioritize Bug 1024568 higher -- perhaps by enlisting additional personnel -- rather than investing the engineering resources required to tack it on to user feedback.

Comment 20

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm not sure I can weigh in on comment #5 since I lack the legal training and perspective. However, I don't see any answers to Mika's questions in comment #6.

I'm not sure I see anything weigh-in-able on comment #12. Seems like it's a general statement of truth, but there's nothing specific in it about this situation.

Comment 21

[:shell escalante]

Based on complexity - investing in  Bug 1024568  instead at this time.

Comment 22

[:shell escalante]

not collecting automatically as feedback form going away.  there's other bugs for pkerr's work and sucking that in....that are more relevant.