1329034 - Consider using a longer default timeout for some tests on some platforms

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

Timeouts make up a large percentage of frequent intermittent test failures. Some (recently it seems like many) of those timeouts are in mochitest-bc and mochitest-dt on linux32/debug. We expect debug builds to perform significantly slower than opt builds, yet we use the same default timeout. Is that appropriate? Would it be better to use a larger default timeout on debug, or on linux32/debug? Would it be confusing?

See https://bugzilla.mozilla.org/show_bug.cgi?id=1253816#c23 and related for some discussion around a particular example. The default response to a timeout due to a platform-specific long-running test seems to be requestLongerTimeout().

We already increased the default timeout for asan mochitest-bc tests in bug 1313187.

We recently stopped running mochitest-dt on linux32/debug in bug 1328915.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: increase default timeout for browser mochitests to 90 seconds, on debug

The timeout increase for browser-flavored mochitests on asan seemed to help reduce unexpected timeouts on asan. I think it makes sense to do the same for debug builds.

Comparing (no change) https://treeherder.mozilla.org/#/jobs?repo=try&revision=1dc3320a1a940bcf10ab323e0cec403555eb4c32 to (increased timeout on debug) https://treeherder.mozilla.org/#/jobs?repo=try&revision=07e47f1de223643200b9a46f3ec4ce895c6e1cf0 there's not much difference, but I think there is a positive effect on bug 1316243 and bug 1310364.

Comment 2

[Geoff Brown [:gbrown]]

Leave-open since I may make a few timeout adjustments here over time.

Comment 3

[Mark Hammond [:markh] [:mhammond]]

Can we do something similar for xpcshell tests too?

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/20d71b8cba15
Increase default timeout on debug browser mochitests; r=jmaher

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/20d71b8cba15

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Mark Hammond [:markh] from comment #3)
> Can we do something similar for xpcshell tests too?

It looks to me like the default xpcshell timeout is 300 seconds, which seems like a long time already. Do you have some examples where that's too short on Debug?

Comment 7

[Mark Hammond [:markh] [:mhammond]]

A quick look finds a number of such bugs - some are old and related to b2g, but others aren't.

bug 1236897
bug 1212395
bug 1144499
bug 1081128
bug 1081128
http://searchfox.org/mozilla-central/rev/f8c9d72cfdaede3e443b814e2d521b650b1d3de6/toolkit/components/osfile/tests/xpcshell/xpcshell.ini#4

IIUC, all the above are timeout failures only on debug builds.

FWIW, http://searchfox.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#671 is where a default timeout is specified and indicates how it can be overridden (although I suspect this is not well known, so many people just split tests in this case)

That said though, I must agree the problem seems far less common with xpcshell, so I'm happy to drop this.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/9711dc9b0835

Comment 9

[Geoff Brown [:gbrown]]

I have been watching Orange Factor for additional time out problems that might be addressable here, and haven't found any.  There are lots of "Test timed out" failures (mostly mochitest), but most of those seem to be hangs, rather than long-running tests that run out of time.

I appreciate the xpcshell suggestion - comment 7 especially - but I am not comfortable increasing the default timeout because 300 seconds is already a long time (consider time wasted for hangs) and I don't see a lot of troublesome long-running xpcshell tests: I'd prefer to handle those long-running cases by splitting tests or using requestlongertimeout on a case-by-case basis.

I feel good about the change we made for browser mochitests on debug platforms: that made a significant, long-lasting difference to failure rates.

Comment 13

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.