1570089 - Recover when replaying processes crash

Status: RESOLVED FIXED

(Core Graveyard :: Web Replay, enhancement)

Description

[Brian Hackett [Laid off!]]

Attachment: patch

If a replaying process crashes, we should be able to recover and continue the debugging session with little interruption for the user. There used to be a mechanism for this, but it was removed as it could lead to long delays for users --- when a process crashed, we would start a new one and bring it to the exact same point as the old one before the user could resume interacting with the tab. With the new control architecture, it is possible to do much better here. Most of the time, replaying processes are running tasks asynchronously in the background; if one of them crashes, we start a new one and the new set of processes will continue running tasks and sharing duties. Sometimes the crashed process is one which is being brought to a point where the tab is paused, but in this case we can just pick another process (whichever is closest to the target point) to take over that job.

The attached patch adds this functionality, though for simplicity there are a couple cases in which a replaying process crash will bring down the whole tab. This patch also fixes some other bugs around crash reporting:

• If a recording process crashes, the middleman was not getting notified and didn't exit, leaving the tab in an unresponsive state instead of a crashed one.

• If a replaying process hangs while processing an asynchronous task, we never detect the hang and the tab could end up in an unresponsive state when e.g. resuming.

Comment 1

[Brian Hackett [Laid off!]]

Attachment: Bug 1570089 Part 1 - Recover from replaying process crashes, r=loganfsmyth.

Comment 2

[Brian Hackett [Laid off!]]

Attachment: Bug 1570089 Part 2 - Associate replaying process crash reports with their own pid, r=gsvelto.

Depends on D39925

Comment 3

[Brian Hackett [Laid off!]]

Attachment: Bug 1570089 Part 3 - Generate replaying process crash reports when requested by the middleman, r=mccr8.

Depends on D39926

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/73faa9185b0d
Part 1 - Recover from replaying process crashes, r=loganfsmyth.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/28b46b8166ee
Part 2 - Associate replaying process crash reports with their own pid, r=gsvelto.
https://hg.mozilla.org/integration/autoland/rev/6b5eb4429f83
Part 3 - Generate replaying process crash reports when requested by the middleman, r=mccr8.

Comment 6

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/73faa9185b0d
https://hg.mozilla.org/mozilla-central/rev/28b46b8166ee
https://hg.mozilla.org/mozilla-central/rev/6b5eb4429f83