1480061 - Webdriver tests in CI don't use long timeout even if specified

Status: RESOLVED INVALID

(Testing :: web-platform-tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

While working on bug 1479850 I noticed more and more timeouts of tests for the wdspec test job. All of those aren't clear to me because of strange timeout settings as visible in the CI produced log files.

(In reply to Henrik Skupin (:whimboo) from bug 1479850 comment #3)
> Having a look at the try results the close window tests are timing out
> pretty often for debug and QuantumRender builds:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=b9afb4aea811023b03236ae23bcda3cabc3f4435
> 
> What's strange is the used timeout until the test got aborted with a timeout
> error. For opt builds it's set to 30, and for debug builds to 80. That's not
> what we have defined as default for wdspec:
> 
> https://searchfox.org/mozilla-central/rev/
> 196560b95f191b48ff7cba7c2ba9237bba6b5b6a/testing/web-platform/tests/tools/
> wptrunner/wptrunner/wpttest.py#424
> 
> Given that all the tests are marked with `# Meta: timeout=long` I would
> expect that tests can take up to 180s. I'm even not sure what the difference
> is between opt/debug and which multiplier we make use of here. So it might
> even be longer.

So I added some debug lines to wpttest.py which log the timeout settings. Here is what I got:

1) Run with `mach wpt --webdriver-binary target/debug/geckodriver --webdriver-arg=-vv --no-capture testing/web-platform/tests/webdriver/tests/` locally

***** TEST TIMEOUT: 180
***** TIMEOUT MULTIPLIER: 3
***** EXTRA TIMEOUT: 5
***** USED TIMEOUT: 545
========================
***** PYTEST TIMEOUT: 545

2) Run in CI via a try build: https://treeherder.mozilla.org/logviewer.html#?job_id=191318647&repo=try&lineNumber=26744-26752

***** TEST TIMEOUT: 25
***** TIMEOUT MULTIPLIER: 3
***** EXTRA TIMEOUT: 5
***** USED TIMEOUT: 80
***** PYTEST TIMEOUT: 80

Both are the timeouts for debug builds, which show a huge difference of the value as used for pytest. Please note the timeout default value which differs here. Checking the defaults in wpttest.py you see the following:

>    default_timeout = 25
>    long_timeout = 180  # 3 minutes

So it means that even a test has a long timeout defined via `# META: timeout=long` it's not taken into account. But that only in CI!

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It was actually my fault. I missed to run `mach wpt --manifest-update` so the extended timeout was not written to MANIFEST.json. :(