Closed Bug 712828 Opened 12 years ago Closed 12 years ago

Mark slow tests as slow

Categories

(Core :: JavaScript Engine, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla12

People

(Reporter: terrence, Assigned: terrence)

References

Details

Attachments

(1 file)

On the js-test testsuite commandline, we can set a per-test timeout, -t N, to kill a test if it takes more than N seconds.  If we vary the per-test timeout and see how long the tests take overall to finish and how many tests we caused to fail because of the timeout, an interesting result pops out of the data:

Timeout |  Test Time | # Failed
--------------------------------
   5    |    37.2    |    11
  10    |    42.4    |     4
  20    |    52.7    |     4
  30    |    60.8    |     5
  40    |    73.4    |     4
  50    |    78.0    |     5
  60    |    88.0    |     4
  70    |    98.7    |     4
default |   123.0    |     0

The result is clear: 4 tests are making our test suite take 3x longer than it otherwise would.  At 40 seconds, the suite is pretty useful, at two minutes, very much less so.  This problem is exacerbated disproportionately in a debug build. 

The four slow tests are:
    js1_5/extensions/regress-336409-2.js
    js1_5/extensions/regress-342960.js
    js1_5/extensions/regress-336410-2.js
    js1_5/extensions/regress-336410-1.js

Note: the 5th failure is the same flaky test in every instance.  I will file a different bug for it.
Depends on: 712519, 712793
Yes, please!
This simply adds the "slow" flag to these 4 tests in the test manifest.  This appears to just work when running tests from the command line.  I am not sure what the implications are for TBPL, but I can't see this making things worse.
Attachment #584637 - Flags: review?(dmandelin)
Attachment #584637 - Flags: review?(dmandelin) → review+
Has anything gotten slower recently, or have these always been slow? If there was a change, we should try to figure out what caused it.
https://hg.mozilla.org/mozilla-central/rev/f8d8caf0a60a
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla12
(In reply to Bill McCloskey (:billm) from comment #4)
> Has anything gotten slower recently, or have these always been slow? If
> there was a change, we should try to figure out what caused it.

I do not know off hand.  According to the wiki, the js test suite should run in ~30 seconds, so presumably when that was written, these tests were fast or not present.  On the other hand, the test suite has taken several minutes to run for as long as I can remember -- I just learned of the timeout hack recently.  Do we have nightly builds of the shell that I could test against?
You need to log in before you can comment on or make changes to this bug.