Last Comment Bug 712828 - Mark slow tests as slow
: Mark slow tests as slow
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: Trunk
: All All
: -- normal (vote)
: mozilla12
Assigned To: Terrence Cole [:terrence]
:
Mentors:
Depends on: 712519 712793
Blocks:
  Show dependency treegraph
 
Reported: 2011-12-21 17:05 PST by Terrence Cole [:terrence]
Modified: 2011-12-29 10:48 PST (History)
4 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
v1: Mark tests as slow. (2.94 KB, patch)
2011-12-28 14:05 PST, Terrence Cole [:terrence]
dmandelin: review+
Details | Diff | Review

Description Terrence Cole [:terrence] 2011-12-21 17:05:15 PST
On the js-test testsuite commandline, we can set a per-test timeout, -t N, to kill a test if it takes more than N seconds.  If we vary the per-test timeout and see how long the tests take overall to finish and how many tests we caused to fail because of the timeout, an interesting result pops out of the data:

Timeout |  Test Time | # Failed
--------------------------------
   5    |    37.2    |    11
  10    |    42.4    |     4
  20    |    52.7    |     4
  30    |    60.8    |     5
  40    |    73.4    |     4
  50    |    78.0    |     5
  60    |    88.0    |     4
  70    |    98.7    |     4
default |   123.0    |     0

The result is clear: 4 tests are making our test suite take 3x longer than it otherwise would.  At 40 seconds, the suite is pretty useful, at two minutes, very much less so.  This problem is exacerbated disproportionately in a debug build. 

The four slow tests are:
    js1_5/extensions/regress-336409-2.js
    js1_5/extensions/regress-342960.js
    js1_5/extensions/regress-336410-2.js
    js1_5/extensions/regress-336410-1.js

Note: the 5th failure is the same flaky test in every instance.  I will file a different bug for it.
Comment 1 Luke Wagner [:luke] 2011-12-21 17:30:22 PST
Yes, please!
Comment 2 Terrence Cole [:terrence] 2011-12-28 14:05:32 PST
Created attachment 584637 [details] [diff] [review]
v1: Mark tests as slow.

This simply adds the "slow" flag to these 4 tests in the test manifest.  This appears to just work when running tests from the command line.  I am not sure what the implications are for TBPL, but I can't see this making things worse.
Comment 3 Terrence Cole [:terrence] 2011-12-28 14:53:52 PST
https://hg.mozilla.org/integration/mozilla-inbound/rev/f8d8caf0a60a
Comment 4 Bill McCloskey (:billm) 2011-12-28 21:05:31 PST
Has anything gotten slower recently, or have these always been slow? If there was a change, we should try to figure out what caused it.
Comment 5 Marco Bonardo [::mak] 2011-12-29 03:34:42 PST
https://hg.mozilla.org/mozilla-central/rev/f8d8caf0a60a
Comment 6 Terrence Cole [:terrence] 2011-12-29 10:48:45 PST
(In reply to Bill McCloskey (:billm) from comment #4)
> Has anything gotten slower recently, or have these always been slow? If
> there was a change, we should try to figure out what caused it.

I do not know off hand.  According to the wiki, the js test suite should run in ~30 seconds, so presumably when that was written, these tests were fast or not present.  On the other hand, the test suite has taken several minutes to run for as long as I can remember -- I just learned of the timeout hack recently.  Do we have nightly builds of the shell that I could test against?

Note You need to log in before you can comment on or make changes to this bug.