1438929 - Change the crash reporter UI to encourage users to describe steps to reproduce

Status: NEW

(Toolkit :: Crash Reporting, defect)

Description

[Jonathan Watt [:jwatt]]

We have a real problem with crash reports that we can't reproduce. A lot of scarce developer time is wasted trying to work out steps to reproduce, or come up with and attempt to fix exotic scenarios under which the code might conceivably be broken (often unsuccessfully).

Looking at the "Gah! Your tab just crashed" UI it's really not surprising that users aren't describing what causes their crashes. Fixing this could save us a bunch of time and reduce user crashes.

Comment 1

[Jonathan Watt [:jwatt]]

Attachment: Annotated issues with current UI

Comment 2

[Jonathan Watt [:jwatt]]

Attachment: Suggested new UI (or something like it)

Comment 3

[Jonathan Watt [:jwatt]]

Setting tracking flag to try and get some visibility for this bug since I think it's important.

Comment 4

[(not currently active) Ted Mielczarek]

All of these UIs have gone through many revisions, so I don't think there's any perfect solution here. I'm extremely skeptical that *any* changes will have any noticeable impact on the amount of useful comments we receive. When someone's browser (or tab) crashes they're frustrated and they want to get back to what they were doing. They aren't likely to care about helping us fix it.

Comment 5

[Jonathan Watt [:jwatt]]

The UI has gone through many revisions. But I've never seen a revision that:

 * told the user their actions leading up to their crash might
   be important to us diagnosing it

 * was sufficiently succinct for such a message to have
   much of a chance of being read

That seems critical to me. Only a small minority of users will provide info, but if they're not actually hearing us ask for it we can hardly be surprised we don't get much.

We'd often only need the odd one in a few hundred users to provide a useful comment. People who repeatedly encounter the same crash (the people most likely to have STRs) are particularly likely to be willing to provide info to help resolve it, as evidenced by the many people looking for help with Firefox crashes on SUMO and elsewhere. Succinctly asking for this info seems worth a shot to me...

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

It sounds like a great hypothesis for an experiment!

Comment 7

[Jonathan Watt [:jwatt]]

shorlander took a cursory look and asked to be needinfo'ed.

Comment 8

[Wayne Mery (:wsmwk)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> All of these UIs have gone through many revisions, so I don't think there's
> any perfect solution here. I'm extremely skeptical that *any* changes will
> have any noticeable impact on the amount of useful comments we receive. When
> someone's browser (or tab) crashes they're frustrated and they want to get
> back to what they were doing. They aren't likely to care about helping us
> fix it.

Sure, people get frustrated. But I've had enough direct contact with crash reporters to say it simply isn't true that none pay attention to the UI or don't care. With millions of users, you don't need a huge percentage of users to make a difference in the outcome.  Many I contact are keen to help - perhaps even more likely to help if they crash frequently - which is great because they might be able to provide steps to reproduce.  

And an additional issue which I'm not sure can be addressed here - many think it's good enough to just report one or two crashes and stop reporting.  

It surely cannot hurt to attempt improvements.

Comment 9

[Stephen Horlander [:shorlander]]

Summary from meeting:

- This UI *has* gone through many revisions
-- This may be a contributing factor to the current visual state
-- It could be streamlined

- Things we need to balance:
-- Users are frustrated and probably want to get back on task
-- Fixing crashes has important user benefit and more info could help fix crashers
-- Privacy and Security controls around what information people are submitting
-- Surfacing areas of information collection that help us fix crashes while still allowing users easy access to get back to their task

I don't see a lot of risk in experimenting with this page to see if we can increase responses while still retaining the primary user need of getting back on task.

Michelle is going to take a look at the content and see what kinds of improvements can be made.

Comment 10

[(not currently active) Ted Mielczarek]

As long as we preserve the current "Reload this tab" button + behavior  I'm not overly concerned with the rest of the UX here. (When we designed the original standalone crashreporter UI that was one of my sticking points--I wanted it focused on getting the user back to what they were doing, with "gather crash reporting data" as a secondary goal.)

Comment 11

[Wayne Mery (:wsmwk)]

To the point of "increasing responses", I have two further observations based on almost daily contact with crash users. 

A. They frequently do not send all their crash reports. I don't have deep understanding of why, but some clearly think all their crashes must be the same issue or cause. But I often find they have multiple issues.

B. They frequently do not send their FIRST crash. My guess is these users think they don't have a problem worth until they hit their second or third crash.

In a production release where you have tens or hundreds of millions of users, these two behaviors might not have statistical or practical impact in surfacing important or emerging crashes.  But if your user pool is relatively small, say 50k users for Firefox nightly 60.0a1, I should think these behaviors must have an effect on the speed and clarity of how a crash signature comes to the attention of QA and devs who are monitoring crashes.

Unfortunately I don't have statistics on these points. Do we have metrics or has anyone looked into the statistics?  Or actually studied how user behavior and perception of crash reporter UI?

Comment 12

[Jonathan Watt [:jwatt]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #10)
> I wanted it focused on getting the user back to what they were doing,
> with "gather crash reporting data" as a secondary goal.

Agreed.

Comment 13

[Gabriele Svelto [:gsvelto]]

(In reply to Wayne Mery (:wsmwk) from comment #11)
> In a production release where you have tens or hundreds of millions of
> users, these two behaviors might not have statistical or practical impact in
> surfacing important or emerging crashes.  But if your user pool is
> relatively small, say 50k users for Firefox nightly 60.0a1, I should think
> these behaviors must have an effect on the speed and clarity of how a crash
> signature comes to the attention of QA and devs who are monitoring crashes.

We're embedding stacks from every crash in crash pings since Firefox 58. While we're still not generating signatures from those we should rather soon. Once we have that the number of crash reports we get will be less relevants WRT detecting a particular signature because we should be able to do it directly from telemetry data.