[meta] Loop Client needs Automated Log Collection for User Issues

RESOLVED INCOMPLETE

Status

Hello (Loop)
Client
P4
normal
Rank:
47
RESOLVED INCOMPLETE
4 years ago
a year ago

People

(Reporter: abr, Unassigned)

Tracking

unspecified
Points:
---
Dependency tree / graph
Bug Flags:
firefox-backlog -

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
Currently, to diagnose failed calls, users need to collect significant information (see <https://wiki.mozilla.org/Loop/Logging> for details). This is a rather high barrier to reporting, and most (if not all) of this can be automated.

Subsequent to call failure, as well as during a call, users should have the ability to click on a button that indicates that the call experience is unsatisfactory. This button will collect the required data and (with user consent) upload it to Telemetry for analysis. 

Users should be provided with a report UUID that they can provide to developers for correlation to the submitted report.

We probably also want to consider prompting users to type in an email address that can be used to contact them regarding the report if any additional information is required.

See also https://wiki.mozilla.org/Loop/Telemetry#User_Issue_Reports

Comment 1

4 years ago
It feels it should be part of the user feedback form prompted to the user at the end of the call.
When the user reports a Negative user experience we plan on displaying a list of options the user has to select from to explain why his call was bad ([]Bad audio [] Bad Video []Got disconnected []Horrible UI []Other with text field).

We could also expose a tick box (probably enabled by default) offering to the user to upload his call data as well as a text box for the e-mail.

I'll mark this bug as a blocker for the desktop client feedback form and the Web UI feedback form as I assumed we can also collect stats from third party WebRTC browsers using the stats API.
Blocks: 972992, 974873
(Reporter)

Comment 2

4 years ago
(In reply to Romain Testard [:RT] from comment #1)
> It feels it should be part of the user feedback form prompted to the user at
> the end of the call.

There's a complicating factor involved here: once the call is over, it may be difficult to collect the data we want to. For example, if a user has a 20-minute call, but only the last 20 seconds had unusable media, it would be far more useful to report the stats for the media near the user "this isn't working" feedback rather than capturing media stats for the whole call (which may well be too much data), or reporting aggregate stats for the call (which wouldn't look too bad for a 20 minute call with only 20 seconds of bad quality).

Updated

3 years ago
Target Milestone: --- → mozilla34

Updated

3 years ago
Whiteboard: [test]
Priority: -- → P1
No longer blocks: 974873
Blocks: 1003180
No longer blocks: 1003180
Whiteboard: [test] → [p=1, investigation, test]
(Reporter)

Comment 3

3 years ago
Incidentally, the design for this is nearly done; I plan to have a design completed by the end of this week (i.e., by August 22nd). See https://wiki.mozilla.org/Loop/Data_Collection#User_Issue_Reports
We'd love this ASAP, but this won't be completed until Fx35.
Target Milestone: mozilla34 → mozilla35

Comment 5

3 years ago
Note - this should capture details on the client type  (Link clicker UI on Chrome, Link clicker UI on Firefox, Desktop client UI on Linux, Desktop client UI on Windows, FxOS, ... ) as it is not currently collected from input.mozilla.org. Only OS info is currently available.

Comment 6

3 years ago
Talking with Patrick from Support - when a user has an issue that they can't find in SUMO and they escalate to the forum - they can upload their logs from firefox directly to crash-stats.  crash stats gives them a link that they can put in their forum post publicly.  only internal mozilla folks can access crash stats - so it's nice to not have to exchange email addresses via the forum or otherwise.  

Would it make sense to see about putting the Hello logs gathered to crash-stats page?

Robert Kaiser is your guy for Crash Stats questions.

Email: rkaiser@mozilla.com

We also have a dev group who created or implemented the API to link to the crash report on our forum.

Ricky Rosario would be the guy to talk to for that.

Updated

3 years ago
backlog: --- → Fx36+
Small correction for Comment 6. Crash Stats is public available. I believe the data is anonymized if you're part of the public and the complete data set is available if you log in via LDAP. 

It would be nice if the complete dataset could be available to TokBox support assuming we're able to share this data with partners.

Comment 8

3 years ago
pushing to 38 as we do need data scrubbing, privacy notice update, and potentially a different location if doesn't work.  start conversations now though.
backlog: Fx36+ → Fx38?
Whiteboard: [p=1, investigation, test] → [p=1, investigation, test][blocked on discussions]
Adam gave me feedback that he has already had "data scrubbing" discussions and thinks we have our ducks in a row with regards to privacy on this bug.  So let's push to get this done by Fx37.
backlog: Fx38? → Fx37+

Updated

3 years ago
Duplicate of this bug: 1082933

Comment 11

3 years ago
We had an email discussion and I believe we reached general agreement. I would still like to do a data-collection review of the final changes (a patch which changes the in-tree docs).
Moving this to P2 based on our new priority definitions.
Priority: P1 → P2
Adam -- Can you write up what needs to be done here so that a developer can take this bug and finish it?  (Is there additional work someone has to do before a developer can start coding?)
backlog: Fx37+ → Fx38?
Flags: needinfo?(adam)
(Reporter)

Comment 14

3 years ago
(In reply to Maire Reavy [:mreavy] (Plz needinfo me) from comment #13)
> Adam -- Can you write up what needs to be done here so that a developer can
> take this bug and finish it?  (Is there additional work someone has to do
> before a developer can start coding?)

The overall work description is at https://wiki.mozilla.org/Loop/Telemetry#User_Issue_Reports -- it's done enough that I think we can begin design and implementation. This is kind of a big parcel of work that probably needs to be broken down into a few different parts. I can see a pretty natural division of labor along the lines of:

1) UX design for each of the five use cases described at https://wiki.mozilla.org/Loop/Telemetry#Use_Cases

2) UX design for the user submission box (similar to the crash reporter UI), informing the user about the data they're about to submit, giving them a chance to examine it, and giving them an opportunity to add their email address to the report, if they so desire.

3) Implementation of the UI work called for by the UX design from #1 above (probably five bugs)

4) Implementation of the UI work called for by the UX design from #2 above

5) Collection of the data detailed in https://wiki.mozilla.org/Loop/Telemetry#Data_to_Collect and creation of the index object -- this probably requires tweaks to other parts of the system to allow easy programmatic access to the indicated data, and may require enhancing logging for certain systems (e.g., the push log may need more detail).

6) Submission of the data to Azure (including interacting with the Loop server to acquire a signed Azure upload URL)

7) Report upload retry logic (see https://wiki.mozilla.org/Loop/Telemetry#Report_Throttling)

8) Loop server work to ingest report index objects, upload them to an Azure table, generate and return the Upload URL to the client

9) Client work to show history of generated reports, so user can refer to them by issue ID (probably want to add an "about:loopissues" or something of that sort)

10) Custom, access-controlled dashboard to allow aggregation of reports, searching for reports by index values, searching for reports by ID, and retrieval of uploaded zip file.

Which means that this is really a meta bug -- retitling accordingly.
Flags: needinfo?(adam)
Summary: Loop Client needs Automated Log Collection for User Issues → [meta] Loop Client needs Automated Log Collection for User Issues

Updated

3 years ago
Depends on: 1119590

Updated

3 years ago
Depends on: 1119595

Comment 15

3 years ago
Hi Maire, I made bugs for 1-4 in Adam's break down - but want to talk to you about what component i should file parts 5-10 under.  some sounds like platform, some server, and possibly a few more client.
backlog: Fx38? → -
Flags: needinfo?(mreavy)
Whiteboard: [p=1, investigation, test][blocked on discussions]
Target Milestone: mozilla35 → ---
Hi Shell, I think Adam has the automated log collection requests in bugs now.  If you still need something from me on this, please feel free to needinfo me again.
Flags: needinfo?(mreavy)

Updated

3 years ago
Rank: 47
Flags: firefox-backlog-
Priority: P2 → P4
Sevaan, could you please give us your thoughts on how this could look like on the UI?
Flags: needinfo?(sfranks)
Paul, this is the Hello bug for collection of user issues. You mentionned some WebRTC work to package call related data - could you please point us to the bug describing this so we can understand if this is something we should re-use for the purpose of this bug?
Flags: needinfo?(pkerr)

Comment 19

2 years ago
The bug is 1156313.
Flags: needinfo?(pkerr)

Updated

2 years ago
Depends on: 1156313
Created attachment 8628943 [details]
Hello Feedback Reporter.png

As part of the visual refresh, there will be a gear icon in the conversation window (accessible at any time/stage of a call) with a sub-option to Submit Feedback (http://i.sevaan.com/image/3K2q1w3A2X0A). If this triggered a window similar to the crash reporter UI, would that work here?

Attached is a mockup.
Flags: needinfo?(sfranks)
Sevaan, it seems like unless people already knew about it, there might be discoverability issues since people are going to want to do this (and we'll want them to do it) most especially when they're having call problems.  

Perhaps there are ways to surface this more directly when the client can tell there has been a problem?  Eg failed call, packet loss above some threshold, etc?
Is the mechanism okay though? We could easily add a "Send Mozilla diagnostic information" to our error screen and call-ended screens to make it more visible.
Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.