1414063 - Adjust mochitest and reftest default output timeouts

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

This is a follow-up to https://bugzilla.mozilla.org/show_bug.cgi?id=1397201#c64.


In browser tests, under normal conditions:

 - We want the js test harness to detect and handle normal test hangs.

 - We want marionette to detect and handle hangs on a marionette socket operation.

 - If a test still does not produce output despite those safeguards, we want the python test harness to time out.


Currently the js mochitest and reftest timeouts are 300 seconds:

https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/TestRunner.js#91

https://dxr.mozilla.org/mozilla-central/rev/b5a3b8ef6902998507fc881b6d628b055457fe31/layout/tools/reftest/reftestcommandline.py#74


Currently the default marionette socket timeout is 360 seconds:

https://dxr.mozilla.org/mozilla-central/rev/b5a3b8ef6902998507fc881b6d628b055457fe31/testing/marionette/client/marionette_driver/marionette.py#562


Currently the default python mochitest and reftest timeouts are 330 seconds:

https://dxr.mozilla.org/mozilla-central/rev/b5a3b8ef6902998507fc881b6d628b055457fe31/testing/mochitest/runtests.py#2655

https://dxr.mozilla.org/mozilla-central/rev/b5a3b8ef6902998507fc881b6d628b055457fe31/layout/tools/reftest/runreftest.py#801


Let's change the default python timeouts to > 360 seconds.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: increase default mochitest/reftest python harness output timeout to 370 seconds

https://treeherder.mozilla.org/#/jobs?repo=try&revision=da14f67172e64edcab0a803986c8d1d6bdd16605

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8924747 [details] [diff] [review]
increase default mochitest/reftest python harness output timeout to 370 seconds

Review of attachment 8924747 [details] [diff] [review]:
-----------------------------------------------------------------

do we have to worry about android in a different fashion?  What about mozharness- does it have a timeout on the output handler?

::: layout/tools/reftest/runreftest.py
@@ +797,5 @@
> +                # The default Marionette socket timeout is currently 360 seconds.
> +                # Wait a little (10 seconds) more before timing out here.
> +                timeout = options.timeout + 30.0
> +                if timeout < 360:
> +                    timeout = 370

assuming we set options.timeout, this would be confusing for people- can you output a warning message to inform the end user?  Also who really uses options.timeout?

Comment 3

[Geoff Brown [:gbrown]]

In my try push, there is at least one "370 seconds without output":

https://treeherder.mozilla.org/logviewer.html#?job_id=141822444&repo=try

...so maybe this won't help bug 1397201?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

if we got to 370, does that mean that marionette completed?

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2)
> What about
> mozharness- does it have a timeout on the output handler?

mozharness typically runs the test harness with an output timeout of 1000 seconds.

https://treeherder.mozilla.org/logviewer.html#?job_id=141822444&repo=try&lineNumber=1941

22:53:35     INFO - Calling ['Z:\\task_1509661616\\build\\venv\\Scripts\\python', '-u', 'Z:\\task_1509661616\\build\\tests\\mochitest\\runtests.py', '--total-chunks', '8', '--this-chunk', '1', '--appname=Z:\\task_1509661616\\build\\application\\firefox\\firefox.exe', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=Z:\\task_1509661616\\build\\symbols', '--certificate-path=tests/certs', '--quiet', '--log-raw=Z:\\task_1509661616\\build\\blobber_upload_dir\\mochitest-devtools-chrome-chunked_raw.log', '--log-errorsummary=Z:\\task_1509661616\\build\\blobber_upload_dir\\mochitest-devtools-chrome-chunked_errorsummary.log', '--screenshot-on-fail', '--cleanup-crashes', '--marionette-startup-timeout=180', '--flavor=browser', '--subsuite=devtools', '--chunk-by-runtime'] with output_timeout 1000

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2)
> do we have to worry about android in a different fashion?

I think Android is okay. Android mochitests and Android opt reftests use the same timeout as desktop. Android debug reftests set options.timeout to 600. (Also, Android mochitests and reftests don't use marionette.)

Comment 7

[Geoff Brown [:gbrown]]

More runs with windows 10 debug mochitests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1ac8f59b69be797466e9e1251ce18ec7005d3318

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Geoff Brown [:gbrown] from comment #5)
> (In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2)
> > What about
> > mozharness- does it have a timeout on the output handler?
> 
> mozharness typically runs the test harness with an output timeout of 1000
> seconds.

So when mozharness typically uses 1000s for output timeout, why are mochitests run with 330s? Looks like we miss to specify the 1000s, or do not correctly forward this value to the mochitest harness?

Comment 9

[Geoff Brown [:gbrown]]

We don't forward the mozharness output timeout to the mochitest harness. I've always thought of them as separate, with one as a backup for the other: If the mochitest harness correctly times out, the mozharness timeout is never hit; if the mochitest harness somehow fails to time out and complete, the mozharness timeout should still work.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8924747 [details] [diff] [review]
increase default mochitest/reftest python harness output timeout to 370 seconds

Review of attachment 8924747 [details] [diff] [review]:
-----------------------------------------------------------------

I would leave this review for Joel given that he has better knowledge about both harnesses.

Comment 11

[Geoff Brown [:gbrown]]

Attachment: increase default mochitest/reftest python harness output timeout to 370 seconds

This is hard to make pretty, but here's a slightly different way of getting to the same place in most cases. Better?

(Regardless of the structures and details, keep in mind that try pushes indicate this will not reduce timeouts, nor improve diagnostics on timeout. On the other hand, it seems logical to keep the harness output timeout > the marionette socket timeout.)

Comment 12

[Geoff Brown [:gbrown]]

I also changed "5 * 60" to "300" since that's what I always search for and can't find!

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8925106 [details] [diff] [review]
increase default mochitest/reftest python harness output timeout to 370 seconds

Review of attachment 8925106 [details] [diff] [review]:
-----------------------------------------------------------------

lets go with it

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1efa6f4269ca
Increase mochitest/reftest default output timeout to 370 seconds; r=jmaher

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7ee2f9d5bdcc
Follow-up for lint errors

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/1efa6f4269ca
https://hg.mozilla.org/mozilla-central/rev/7ee2f9d5bdcc