Closed Bug 1329034 Opened 3 years ago Closed 3 years ago

Consider using a longer default timeout for some tests on some platforms

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(1 file)

Timeouts make up a large percentage of frequent intermittent test failures. Some (recently it seems like many) of those timeouts are in mochitest-bc and mochitest-dt on linux32/debug. We expect debug builds to perform significantly slower than opt builds, yet we use the same default timeout. Is that appropriate? Would it be better to use a larger default timeout on debug, or on linux32/debug? Would it be confusing?

See https://bugzilla.mozilla.org/show_bug.cgi?id=1253816#c23 and related for some discussion around a particular example. The default response to a timeout due to a platform-specific long-running test seems to be requestLongerTimeout().

We already increased the default timeout for asan mochitest-bc tests in bug 1313187.

We recently stopped running mochitest-dt on linux32/debug in bug 1328915.
See Also: → 1316243
See Also: → 1310364
The timeout increase for browser-flavored mochitests on asan seemed to help reduce unexpected timeouts on asan. I think it makes sense to do the same for debug builds.

Comparing (no change) https://treeherder.mozilla.org/#/jobs?repo=try&revision=1dc3320a1a940bcf10ab323e0cec403555eb4c32 to (increased timeout on debug) https://treeherder.mozilla.org/#/jobs?repo=try&revision=07e47f1de223643200b9a46f3ec4ce895c6e1cf0 there's not much difference, but I think there is a positive effect on bug 1316243 and bug 1310364.
Attachment #8827683 - Flags: review?(jmaher)
Leave-open since I may make a few timeout adjustments here over time.
Keywords: leave-open
Can we do something similar for xpcshell tests too?
Attachment #8827683 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/20d71b8cba15
Increase default timeout on debug browser mochitests; r=jmaher
(In reply to Mark Hammond [:markh] from comment #3)
> Can we do something similar for xpcshell tests too?

It looks to me like the default xpcshell timeout is 300 seconds, which seems like a long time already. Do you have some examples where that's too short on Debug?
A quick look finds a number of such bugs - some are old and related to b2g, but others aren't.

bug 1236897
bug 1212395
bug 1144499
bug 1081128
bug 1081128
http://searchfox.org/mozilla-central/rev/f8c9d72cfdaede3e443b814e2d521b650b1d3de6/toolkit/components/osfile/tests/xpcshell/xpcshell.ini#4

IIUC, all the above are timeout failures only on debug builds.

FWIW, http://searchfox.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#671 is where a default timeout is specified and indicates how it can be overridden (although I suspect this is not well known, so many people just split tests in this case)

That said though, I must agree the problem seems far less common with xpcshell, so I'm happy to drop this.
I have been watching Orange Factor for additional time out problems that might be addressable here, and haven't found any.  There are lots of "Test timed out" failures (mostly mochitest), but most of those seem to be hangs, rather than long-running tests that run out of time.

I appreciate the xpcshell suggestion - comment 7 especially - but I am not comfortable increasing the default timeout because 300 seconds is already a long time (consider time wasted for hangs) and I don't see a lot of troublesome long-running xpcshell tests: I'd prefer to handle those long-running cases by splitting tests or using requestlongertimeout on a case-by-case basis.

I feel good about the change we made for browser mochitests on debug platforms: that made a significant, long-lasting difference to failure rates.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Duplicate of this bug: 1292103
Duplicate of this bug: 1315958
Duplicate of this bug: 1320260
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.