Mark slow tests as slow

RESOLVED FIXED in mozilla12

Status

()

Core
JavaScript Engine
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: terrence, Assigned: terrence)

Tracking

Trunk
mozilla12
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

5 years ago
On the js-test testsuite commandline, we can set a per-test timeout, -t N, to kill a test if it takes more than N seconds.  If we vary the per-test timeout and see how long the tests take overall to finish and how many tests we caused to fail because of the timeout, an interesting result pops out of the data:

Timeout |  Test Time | # Failed
--------------------------------
   5    |    37.2    |    11
  10    |    42.4    |     4
  20    |    52.7    |     4
  30    |    60.8    |     5
  40    |    73.4    |     4
  50    |    78.0    |     5
  60    |    88.0    |     4
  70    |    98.7    |     4
default |   123.0    |     0

The result is clear: 4 tests are making our test suite take 3x longer than it otherwise would.  At 40 seconds, the suite is pretty useful, at two minutes, very much less so.  This problem is exacerbated disproportionately in a debug build. 

The four slow tests are:
    js1_5/extensions/regress-336409-2.js
    js1_5/extensions/regress-342960.js
    js1_5/extensions/regress-336410-2.js
    js1_5/extensions/regress-336410-1.js

Note: the 5th failure is the same flaky test in every instance.  I will file a different bug for it.
(Assignee)

Updated

5 years ago
Depends on: 712519, 712793

Comment 1

5 years ago
Yes, please!
(Assignee)

Comment 2

5 years ago
Created attachment 584637 [details] [diff] [review]
v1: Mark tests as slow.

This simply adds the "slow" flag to these 4 tests in the test manifest.  This appears to just work when running tests from the command line.  I am not sure what the implications are for TBPL, but I can't see this making things worse.
Attachment #584637 - Flags: review?(dmandelin)
Attachment #584637 - Flags: review?(dmandelin) → review+
(Assignee)

Comment 3

5 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/f8d8caf0a60a
Has anything gotten slower recently, or have these always been slow? If there was a change, we should try to figure out what caused it.
https://hg.mozilla.org/mozilla-central/rev/f8d8caf0a60a
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla12
(Assignee)

Comment 6

5 years ago
(In reply to Bill McCloskey (:billm) from comment #4)
> Has anything gotten slower recently, or have these always been slow? If
> there was a change, we should try to figure out what caused it.

I do not know off hand.  According to the wiki, the js test suite should run in ~30 seconds, so presumably when that was written, these tests were fast or not present.  On the other hand, the test suite has taken several minutes to run for as long as I can remember -- I just learned of the timeout hack recently.  Do we have nightly builds of the shell that I could test against?
You need to log in before you can comment on or make changes to this bug.