On the js-test testsuite commandline, we can set a per-test timeout, -t N, to kill a test if it takes more than N seconds. If we vary the per-test timeout and see how long the tests take overall to finish and how many tests we caused to fail because of the timeout, an interesting result pops out of the data:
Timeout | Test Time | # Failed
5 | 37.2 | 11
10 | 42.4 | 4
20 | 52.7 | 4
30 | 60.8 | 5
40 | 73.4 | 4
50 | 78.0 | 5
60 | 88.0 | 4
70 | 98.7 | 4
default | 123.0 | 0
The result is clear: 4 tests are making our test suite take 3x longer than it otherwise would. At 40 seconds, the suite is pretty useful, at two minutes, very much less so. This problem is exacerbated disproportionately in a debug build.
The four slow tests are:
Note: the 5th failure is the same flaky test in every instance. I will file a different bug for it.
Created attachment 584637 [details] [diff] [review]
v1: Mark tests as slow.
This simply adds the "slow" flag to these 4 tests in the test manifest. This appears to just work when running tests from the command line. I am not sure what the implications are for TBPL, but I can't see this making things worse.
Has anything gotten slower recently, or have these always been slow? If there was a change, we should try to figure out what caused it.
(In reply to Bill McCloskey (:billm) from comment #4)
> Has anything gotten slower recently, or have these always been slow? If
> there was a change, we should try to figure out what caused it.
I do not know off hand. According to the wiki, the js test suite should run in ~30 seconds, so presumably when that was written, these tests were fast or not present. On the other hand, the test suite has taken several minutes to run for as long as I can remember -- I just learned of the timeout hack recently. Do we have nightly builds of the shell that I could test against?