479518 - Increase the reftest timeout, at least for tests we know are slow

Status: RESOLVED DUPLICATE of bug 471579

(Testing :: Reftest, enhancement)

Description

[Jesse Ruderman]

<ted> i think we need to declare war on intermittent tests
<Jesse> i think we should increase the reftest timeout
<Jesse> to like 10 minutes
<ted> just because you like to write long-running tests? :)
<Jesse> so we can deal with slow tests out of band, not by having them fail intermittently
<ted> what if we let reftests specify their own timeouts, or something?
<ted> or at least say "i'm a long-running test, give me more time"
<Jesse> i could go for that

Allowing a test to be slow is more efficient than splitting it into multiple smaller tests, each of which has its own overhead.

Comment 1

[Jeff Walden [:Waldo]]

(Not to mention things like bug 473911 really *can't* be split up!)

Comment 2

[Jesse Ruderman]

Attachment: increase reftest timeout from 10 seconds to 10 minutes [Superseded by bug 471579]

Intermittent orange is not the right way to avoid having tests that are too slow.

Comment 3

[Jesse Ruderman]

See also bug 471579, which has a patch to make the timeout vary by platform.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

From the patch I have in bug 471579, you can adjust the timeout in the command line to runreftest.py.  Unless there is a need for this being hardcoded to 10 minutes, we should stick with a flexible solution.

Comment 5

[Jesse Ruderman]

Flexibility is great, but we also need a sane default, especially since Tinderbox uses the default.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

how about I adjust my patch in bug 471579:
https://bug471579.bugzilla.mozilla.org/attachment.cgi?id=380409

to change:
gTimeoutValue = 10000; //ms

to be similar to your value:
gTimeoutValue = 10 * 60 * 1000; //10 minutes


We would also have to adjust my code (in the same patch) in runreftest.py where it accepts --timeout and currently has a default of 10000.  

Ted, could you offer some insight here.

Comment 7

[Jeff Walden [:Waldo]]

Ten minutes seems a bit long; I can't believe two or three minutes wouldn't be more than adequate.

Comment 8

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Maybe five minutes would be better; the difference between five and ten could be noticeable in fixing tinderbox bustage.  Other than that, it really shouldn't matter.

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 380560 [details] [diff] [review]
increase reftest timeout from 10 seconds to 10 minutes
[Superseded by bug 471579]

r=dbaron