Open Bug 1479202 Opened Last year Updated 10 months ago

[meta] Web Replay: Record/replay all mochitests

Categories

(Core :: Web Replay, enhancement)

enhancement
Not set

Tracking

()

Tracking Status
firefox63 --- affected

People

(Reporter: bhackett, Unassigned)

References

Details

(Keywords: meta)

Using the --save-recordings option, the mochitest harness is able to record all the content processes spawned by the tests it runs.  Web Replay's next major goal is to be able to successfully record and replay all mochitests: record all the content processes without hanging or crashing, replay those recordings to completion, and account for any test failures that happened while recording.  Since we don't support all browser features, eliminating all test failures is not an immediate goal.  This will be great for ensuring stability and will vastly increase the coverage of automated tests for recording/replaying.

This is a tracking bug for fixes needed in pursuit of this goal.
Depends on: 1479339
Depends on: 1479547
Depends on: 1479583
Depends on: 1479585
Depends on: 1479641
Depends on: 1479643
Depends on: 1479644
Depends on: 1479645
Depends on: 1479647
Depends on: 1479650
Depends on: 1479651
Depends on: 1481346
Depends on: 1481350
Depends on: 1481351
Depends on: 1481352
Depends on: 1481353
Depends on: 1483240
Depends on: 1483242
Depends on: 1483248
Depends on: 1483256
Depends on: 1486556
Depends on: 1486562
Depends on: 1486565
Depends on: 1486570
Depends on: 1486573
Depends on: 1486591
Depends on: 1486609
Depends on: 1486616
Depends on: 1486619
So, last weekend I finished going through all the browser mochitests (the tests that run with ./mach mochitest browser, about 4000 tests in total), running them individually with the --save-recordings option and then replaying the generated recordings.  With the fixes that have landed and a few more I still need to file:

- 3485 tests ran successfully: the test passes while recording without failures, timeouts, or crashes, and all its recordings can be replayed to completion.

- 151 tests had failures while recording, but no timeouts or crashes, and their recordings can be replayed to completion.

- 174 tests had timeouts (or just ran forever without ever actually timing out) while recording, but no crashes, and their recordings can be replayed to completion.

- 10 tests crashed while recording.

- 55 tests crashed while replaying.

- 108 tests depend on the Gecko Profiler, which we don't support yet while recording/replaying.

So things are in pretty good shape right now but there are still a lot of failures to investigate and fix.
You need to log in before you can comment on or make changes to this bug.