1003180 - [meta] Client needs to report user feedback

Status: RESOLVED FIXED

(Hello (Loop) :: Client, defect, P1)

Comment 1

[:shell escalante]

Is using Google forms like Talkilla did OK for this again?  Also, what questions do we want to ask?

Comment 2

[Romain Testard [:RT]]

We want an experience integrated into the client UX so that at the end of a call the conversation window prompts the user to provide feedback. This is optional for the user.
I discussed with Darrin who will provide UX.

The UX I proposed:
Screen with Happy or Sad displayed at the end of the call in the conversation window
    If happy clicked, thank the user for the feedback and close conversation window
    If Sad clicked display another screen: "What makes you sad? []Bad audio [] Bad Video []Got disconnected []Horrible UI []Other with text field

This although requires server back-end to store the data and I'd like to understand if this is technically complex or not. We will require this back-end also for the mobile application which can't nicely display a Google form on mobile form factor.

If this is likely to take time to implement this, an intermediary step could be to display a link to a Google form at the end of the call in the convresation window, when the user clicks it a new tab with the form opens. An example of the form is: 
https://docs.google.com/forms/d/1KSr1Zy5SupSxXk-6JE2x3pzRulrd4wo6jbcgCvC3rC0/viewform

I needinfo Mark for comments on whether we have a simple way to store that data which should help understand if we need the intermediate Google form step.

Comment 3

[Mark Banner (:standard8)]

We only did Google forms previously as that was an easy way of getting something set up quickly. We probably need some advice from the metrics team here, though I do wonder if we can re-use the telemetry servers, or there's maybe something else.

Adam - I think you've looked into this more than me, any ideas?

Comment 4

[Adam Roach [:abr]]

(In reply to Mark Banner (:standard8) from comment #3)
> We only did Google forms previously as that was an easy way of getting
> something set up quickly. We probably need some advice from the metrics team
> here, though I do wonder if we can re-use the telemetry servers, or there's
> maybe something else.
> 
> Adam - I think you've looked into this more than me, any ideas?

Sure, I suspect Telemetry would be just fine for this, although I'd want to run it by someone in that group (Chris Reid or Tarek) to make sure they agree in principle. If you look at <https://wiki.mozilla.org/Loop/Telemetry>, you'll see what we're currently collecting for ICE failures. I don't see any reason we shouldn't be able to upload user feedback; something like:

{
    "ver": 1,
    "info": {
      "appUpdateChannel": "default",
      "appBuildID": "20140421104955",
      "appName": "Firefox",
      "appVersion": "32.0",
      "reason": "loop",
      "OS":"Darwin",
      "version":"12.5.0"
    },
    "report": "user feedback",
    "session_id": "...",
    "usability_rating": "6",
    "quality_rating": "8",
    "comments": "...",
    ...
}

It's really just a matter of defining the fields we want to collect here, running them by the privacy folks, and putting together the UX that collects the data.

Comment 5

["Saptarshi Guha[:joy]"]

How will this data structure look per call? If make two calls in a session, and provide feedback for 0,1,2 of those calls, how does the data structure look?

Comment 6

[Adam Roach [:abr]]

(In reply to "Saptarshi Guha[:joy]" from comment #5)
> How will this data structure look per call? If make two calls in a session,
> and provide feedback for 0,1,2 of those calls, how does the data structure
> look?

The intention is that these would be collected on a per-call basis. If you make three calls and then leave feedback, we'll get feedback on the most recently one only.

Comment 7

["Saptarshi Guha[:joy]"]

Thanks Adam. So if I made 3 calls and feedback for all, i'd get 3 pieces of feedback?
Cheers
Saptarshi

Comment 8

[Adam Roach [:abr]]

(In reply to "Saptarshi Guha[:joy]" from comment #7)
> Thanks Adam. So if I made 3 calls and feedback for all, i'd get 3 pieces of
> feedback?

That's the plan, yes.

Comment 9

[:shell escalante]

Hi Saptarshi - does anything need to be ready on the server side for us to land 972992 Desktop feedback and web clicker feedback form 974873?

Comment 10

["Saptarshi Guha[:joy]"]

We (myself and Romain) have a meeting with Matt Grimes of user feedback this Tuesday (15th July).
They will tell us if we need anything else.

Comment 11

[Matt Grimes [:Matt_G]]

(In reply to Romain Testard [:RT] from comment #2)

> The UX I proposed:
> Screen with Happy or Sad displayed at the end of the call in the
> conversation window
>     If happy clicked, thank the user for the feedback and close conversation
> window
>     If Sad clicked display another screen: "What makes you sad? []Bad audio
> [] Bad Video []Got disconnected []Horrible UI []Other with text field
> 

I've had a few conversations with my team since we last met and I wanted to surface a concern. We cannot send reports of abuse (or any other feedback that *requires* a response) to Input. My suggestion would be that we add an option to check boxes above as such:

[] Bad Video 
[] Got disconnected 
[] Horrible UI
[] Report Abuse
[] Other with text field

If the user selects the abuse box, we direct them to the appropriate channel (which is tbd to my knowledge) and we do not submit any information to Input. 

Let me know if this needs further discussion, but I wanted to call this out before we get too far down the road. We need to redirect users to appropriate channels from the product where it makes sense to do so.

Comment 12

[Romain Testard [:RT]]

(In reply to Matt Grimes [:Matt_G] from comment #11)
> (In reply to Romain Testard [:RT] from comment #2)
> 
> > The UX I proposed:
> > Screen with Happy or Sad displayed at the end of the call in the
> > conversation window
> >     If happy clicked, thank the user for the feedback and close conversation
> > window
> >     If Sad clicked display another screen: "What makes you sad? []Bad audio
> > [] Bad Video []Got disconnected []Horrible UI []Other with text field
> > 
> 
> I've had a few conversations with my team since we last met and I wanted to
> surface a concern. We cannot send reports of abuse (or any other feedback
> that *requires* a response) to Input. My suggestion would be that we add an
> option to check boxes above as such:
> 
> [] Bad Video 
> [] Got disconnected 
> [] Horrible UI
> [] Report Abuse
> [] Other with text field
> 
> If the user selects the abuse box, we direct them to the appropriate channel
> (which is tbd to my knowledge) and we do not submit any information to
> Input. 
> 
> Let me know if this needs further discussion, but I wanted to call this out
> before we get too far down the road. We need to redirect users to
> appropriate channels from the product where it makes sense to do so.

Yes I agree. We'll implement the "report user" feature post landing of the feedback form.
When landing the "report user" feature this will probably leverage some client logic to ensure the feedback and "report user" data get stored in a separate place (the "report user" data is anyway subject to data retention policies of 2 years so we'll have to find an appropriate place for this).

So this bug tracks the implementation of the feedback form for the following fields in the desktop client UI (wording as per latest UX available on [1]:
[] Audio quality
[] Video quality 
[] Was disconnected 
[] Confusing
[] Other with text field

Please note taht a link clicker UI bug tracks the feedback form implementation - https://bugzilla.mozilla.org/show_bug.cgi?id=974873 where I added a similar comment.

We're now ready on the UI side and we need more details on how to integrate with the input.mozilla.org back-end - can you please provide the details for this?

[1]
https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/#feedback
Note: click happy and sad faces to see the flow.

Comment 13

[Matt Grimes [:Matt_G]]

Will is our lead Input Developer. He can provide all the API details.

Comment 14

[Will Kahn-Greene [:willkg] ET needinfo? me]

The API for posting data to Input is documented here:

http://fjord.readthedocs.org/en/latest/api.html#posting-product-feedback-api-v1-feedback

Things we/I would need to do before Input is ready for receiving data:

1. We would need to create a product on Input for Loop to post to.

2. Additionally, since you're asking to provide additional details that don't have fields in the API, we need to implement bug #1041622 and bug #1041664.

Neither are hard to do. It's probably a couple of days of work.

Comment 15

[:shell escalante]

Thanks Will, when can the bugs you listed get on the "to do" list and creating a product on Input?  we have the UI ready and the product Loop is out so we'd love feedback :) 

needinfo to Niko on if bug 972992 has everything needed with the API listed below - or if it's blocked on the product in Input and the bugs Will mentioned (which i believe it is).

Comment 16

[Will Kahn-Greene [:willkg] ET needinfo? me]

This comment in this other bug suggests Loop is going to push data to Telemetry for now and y'alls are going to think about pushing to Input at some later date:

https://bugzilla.mozilla.org/show_bug.cgi?id=1041439#c10

So I'm confused as to why you're saying you're all ready to go.

Regarding schedule, I could do this work next week, but I've got a lot of stuff to do and if you're not going to use this now, then I'd rather push it off until it's needed.

Comment 17

[Romain Testard [:RT]]

To be clearer there are 2 parts:
1 User feedback:
[] Audio quality
[] Video quality 
[] Was disconnected 
[] Confusing
[] Other with text field

2 Automated Log collection

[1] is clearly defined and we need now and we want to use input.mozilla.org
[2] is uncertain and we need later

I'm unclear about whether we should use Telemetry or input.mozilla.org for [2] as I don't fully understand the amount of data involved and the structure.
What I suggested is that we leave [2] as a separate item as we need to deliver it with FF35 only as part of our roadmap and we need some input from Adam who is away this week.

I hope this makes sense?

Comment 18

[Will Kahn-Greene [:willkg] ET needinfo? me]

That makes more sense, except for the fact we keep saying Input is good for user sentiment and *not good* for automated log collection.

I'll put those bugs in my queue for next week.

Comment 19

["Saptarshi Guha[:joy]"]

Romain if the log collection is collected separately from feedback it will be difficult to associate the two.

Comment 20

[Romain Testard [:RT]]

(In reply to "Saptarshi Guha[:joy]" from comment #19)
> Romain if the log collection is collected separately from feedback it will
> be difficult to associate the two.

I agree, ideally we should collect the 2 together although we need to confirm (1) what the logs look like (need to confirm with Adam) and then (2) if we could store them on input.mozilla.org.

Given the won't implement the log collection on the client side for another few weeks and we need the basic user feedback collection very soon now (we'll be on Aurora on Friday and really need to start collecting user opinions) it makes sense to handle these in 2 steps it seems.

Comment 21

[Nicolas Perriault (:NiKo`) — needinfo me if you need my attention]

(In reply to sescalante from comment #15)

> needinfo to Niko on if bug 972992 has everything needed with the API listed
> below - or if it's blocked on the product in Input and the bugs Will
> mentioned (which i believe it is).

We definitely need custom fields to match UX, so I think bug 972992 is blocked by bugs 1041622 and/or 1041664 for now. I'll add dependency information and a comment in there.

Comment 22

[Adam Roach [:abr]]

I believe the conclusion we reached yesterday is that we will (a) use input.mozilla.org to store the user-gathered data; and (b) not include gathered metrics for the initial landing.

In terms of custom fields, I think we can go about this a couple of ways. Currently, the only thing in UX that isn't accounted for in input.mozilla.org is the "reason" field. I think making this part of a blob would be kind of odd, since it's something we definitely want to treat as part of the record for the purpose of accounting for *why* people commented the way they did.

Trivially, we could add a "reason" field as an optional parameter to the input API (cf. http://fjord.readthedocs.org/en/latest/api.html).

Alternately, if we would like to deploy without server impact, we could do something like prepending the "description" field with the reason, so it would read something like "Confusing: could not figure out how to hang up". 

I'm tagging Matt here to see which of these he thinks makes the most sense. Matt: the UX we're discussing is described here: https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/#feedback (click on the unhappy face to get to the reasons we're discussing here).

Comment 23

[Will Kahn-Greene [:willkg] ET needinfo? me]

I talked to Matt a bit about the new requirements. We've been talking about adding a category field so that we could do a first pass on classification of incoming feedback. I think I'll implement that now and we can use that field here. I'm going to do a "minimal implementation", so it'll be a string field with a max length of 30 characters. That should cover the issues raised in comment #22.

I wrote up bug #1045942 to add a category field to the response table and the Input API. I'll work on that after I finish the other API work I'm in the middle of for Loop.

If that doesn't cover the needs in comment #22, please let me know.

Comment 24

[Nicolas Perriault (:NiKo`) — needinfo me if you need my attention]

(In reply to Will Kahn-Greene [:willkg] from comment #23)
> it'll be a string field with a max length of 30 characters

This feels a bit short, eg. "could not figure out how to hang up" is 35 chars long :)

Otherwise, sounds good to me.

Comment 25

[Will Kahn-Greene [:willkg] ET needinfo? me]

According to the mockup linked in comment #22 and the list in comment #17, the categories would be:

[] Audio quality
[] Video quality 
[] Was disconnected 
[] Confusing
[] Other with text field

Further, you probably don't want to use the label text and instead use a short representative value string like:

* audio
* video
* disconnected
* confusing
* other

None of those words exceed 30 characters.

Any user-supplied text based input should go in the description field.

Does that clarify?

Comment 26

[Nicolas Perriault (:NiKo`) — needinfo me if you need my attention]

(In reply to Will Kahn-Greene [:willkg] from comment #25)
> Any user-supplied text based input should go in the description field.
> Does that clarify?

Yes, thanks.

Comment 27

[:shell escalante]

Hi Will, is there a bug for a "loop" Product option on Input and the specific Categories mentioned in Comment 25?  I see bugs for setting up input for the capabilities - but wasn't sure if that covered making available to Loop to send to.

your updates on the related bugs are below.  It sounds like only the top 2 are blocking and 1 is heading to production and 1 being worked on now.

bug 1041622 (Will 7/29) "Landed in master in https://github.com/mozilla/fjord/commit/fa0caac1  I'll push it to production tomorrow."

bug #1045942 (Will 7/29) "I'm going to implement this next since I think we're going to use this for Loop."

bug 1041664 (Will 7/29: "Grabbing this one. I think this is a day or two of work. The complexity here is that I have to fiddle with the API stuff which is sometimes hard. Anyhow, we'll see..)"

Comment 28

[Will Kahn-Greene [:willkg] ET needinfo? me]

There's no bug for adding the Loop product--that's not a code change. We'll just do that when we're ready to go.

There's no bug for adding categories because this is just going to be an unverified loose text field. We're not going to be doing foreign key lookups or any other validation of what you're sending.

Comment 29

[:shell escalante]

perfect!  thanks will :).  We are ready now to start submitting, though could we add "Hello" as the Product name instead of Loop (which is code name for the project).

Comment 30

[Will Kahn-Greene [:willkg] ET needinfo? me]

It doesn't really matter what name we use since it won't be publicly visible. So whatever you want is fine.

I pushed all the changes required.

Documentation for the category field are in this section:

http://fjord.readthedocs.org/en/latest/api.html#optional-fields

Documentation for the "extra context" is in this section:

http://fjord.readthedocs.org/en/latest/api.html#extra-context

Comment 31

[Maire Reavy [:mreavy]]

Niko, Dan -- Is this still blocked?  (It's listed as blocked on Trello.)  If you believe this is still blocked, can you give me a brief summary of current status and what needs to happen for this to move forward?     Thanks.

Comment 32

[Nicolas Perriault (:NiKo`) — needinfo me if you need my attention]

(In reply to Maire Reavy [:mreavy] (Plz needinfo me) from comment #31)
> Niko, Dan -- Is this still blocked?  (It's listed as blocked on Trello.)  If
> you believe this is still blocked, can you give me a brief summary of
> current status and what needs to happen for this to move forward?     Thanks.

I don't have a clue why this is blocked on Trello… Maybe Dan knows?

Comment 33

[Adam Roach [:abr]]

I have a proposed set of fields described here: https://wiki.mozilla.org/Loop/Data_Collection#Input_Metrics -- this task should mostly be a matter of implementing the UX (perhaps simply by copying over the react views from Darrin's design) and hooking them up to a trivial XHR POST.

Comment 34

[Dan Mosedale (:dmosedale, :dmose)]

My guess is that this was marked blocked based on the dependent bugs mentioned in comment 27.  Since those are all fixed, I'd guess this could be unblocked.

Comment 35

[Maire Reavy [:mreavy]]

Per our privacy folks, we don't have to report abusive users as a requirement for the first release

Comment 36

[Maire Reavy [:mreavy]]

Updating dependencies per our last team discussion about this bug

Comment 37

[Maire Reavy [:mreavy]]

I believe the requirements for this meta bug have been satisfied.

Comment 38

[u279076]

Does this have sufficient automation coverage or could it use some manual testing?

Comment 39

[u279076]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #38)
> Does this have sufficient automation coverage or could it use some manual
> testing?

Mark, can you please answer this?

Comment 40

[Mark Banner (:standard8)]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #38)
> Does this have sufficient automation coverage or could it use some manual
> testing?

This has been covered enough by the dependent bugs.