1479202 - [meta] Web Replay: Record/replay all mochitests

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Web Replay, enhancement)

Description

[Brian Hackett [Laid off!]]

Using the --save-recordings option, the mochitest harness is able to record all the content processes spawned by the tests it runs.  Web Replay's next major goal is to be able to successfully record and replay all mochitests: record all the content processes without hanging or crashing, replay those recordings to completion, and account for any test failures that happened while recording.  Since we don't support all browser features, eliminating all test failures is not an immediate goal.  This will be great for ensuring stability and will vastly increase the coverage of automated tests for recording/replaying.

This is a tracking bug for fixes needed in pursuit of this goal.

Comment 1

[Brian Hackett [Laid off!]]

So, last weekend I finished going through all the browser mochitests (the tests that run with ./mach mochitest browser, about 4000 tests in total), running them individually with the --save-recordings option and then replaying the generated recordings.  With the fixes that have landed and a few more I still need to file:

- 3485 tests ran successfully: the test passes while recording without failures, timeouts, or crashes, and all its recordings can be replayed to completion.

- 151 tests had failures while recording, but no timeouts or crashes, and their recordings can be replayed to completion.

- 174 tests had timeouts (or just ran forever without ever actually timing out) while recording, but no crashes, and their recordings can be replayed to completion.

- 10 tests crashed while recording.

- 55 tests crashed while replaying.

- 108 tests depend on the Gecko Profiler, which we don't support yet while recording/replaying.

So things are in pretty good shape right now but there are still a lot of failures to investigate and fix.