Last Comment Bug 712828 - Mark slow tests as slow
: Mark slow tests as slow
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: Trunk
: All All
-- normal (vote)
: mozilla12
Assigned To: Terrence Cole [:terrence]
: Jason Orendorff [:jorendorff]
Depends on: 712519 712793
  Show dependency treegraph
Reported: 2011-12-21 17:05 PST by Terrence Cole [:terrence]
Modified: 2011-12-29 10:48 PST (History)
4 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---

v1: Mark tests as slow. (2.94 KB, patch)
2011-12-28 14:05 PST, Terrence Cole [:terrence]
dmandelin: review+
Details | Diff | Splinter Review

Description User image Terrence Cole [:terrence] 2011-12-21 17:05:15 PST
On the js-test testsuite commandline, we can set a per-test timeout, -t N, to kill a test if it takes more than N seconds.  If we vary the per-test timeout and see how long the tests take overall to finish and how many tests we caused to fail because of the timeout, an interesting result pops out of the data:

Timeout |  Test Time | # Failed
   5    |    37.2    |    11
  10    |    42.4    |     4
  20    |    52.7    |     4
  30    |    60.8    |     5
  40    |    73.4    |     4
  50    |    78.0    |     5
  60    |    88.0    |     4
  70    |    98.7    |     4
default |   123.0    |     0

The result is clear: 4 tests are making our test suite take 3x longer than it otherwise would.  At 40 seconds, the suite is pretty useful, at two minutes, very much less so.  This problem is exacerbated disproportionately in a debug build. 

The four slow tests are:

Note: the 5th failure is the same flaky test in every instance.  I will file a different bug for it.
Comment 1 User image Luke Wagner [:luke] 2011-12-21 17:30:22 PST
Yes, please!
Comment 2 User image Terrence Cole [:terrence] 2011-12-28 14:05:32 PST
Created attachment 584637 [details] [diff] [review]
v1: Mark tests as slow.

This simply adds the "slow" flag to these 4 tests in the test manifest.  This appears to just work when running tests from the command line.  I am not sure what the implications are for TBPL, but I can't see this making things worse.
Comment 3 User image Terrence Cole [:terrence] 2011-12-28 14:53:52 PST
Comment 4 User image Bill McCloskey (:billm) 2011-12-28 21:05:31 PST
Has anything gotten slower recently, or have these always been slow? If there was a change, we should try to figure out what caused it.
Comment 5 User image Marco Bonardo [::mak] 2011-12-29 03:34:42 PST
Comment 6 User image Terrence Cole [:terrence] 2011-12-29 10:48:45 PST
(In reply to Bill McCloskey (:billm) from comment #4)
> Has anything gotten slower recently, or have these always been slow? If
> there was a change, we should try to figure out what caused it.

I do not know off hand.  According to the wiki, the js test suite should run in ~30 seconds, so presumably when that was written, these tests were fast or not present.  On the other hand, the test suite has taken several minutes to run for as long as I can remember -- I just learned of the timeout hack recently.  Do we have nightly builds of the shell that I could test against?

Note You need to log in before you can comment on or make changes to this bug.