1469288 - Perma-fail: linux ccov (and linux32 debug and linux64 qr debug) wpt2 | Task timeout after 7200 seconds

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[Geoff Brown [:gbrown]]

Failures are typically reported in bug 1411358.

https://treeherder.mozilla.org/logviewer.html#?job_id=183432805&repo=mozilla-central&lineNumber=34178

[task 2018-06-16T01:54:11.181Z] 01:54:11     INFO - PID 15576 | ++DOCSHELL 0x7f6fd4dd1000 == 11 [pid = 15627] [id = {7a992f67-84c8-4aa0-ae1a-b6a4adaade8b}]
[task 2018-06-16T01:54:11.181Z] 01:54:11     INFO - PID 15576 | ++DOMWINDOW == 40 (0x7f6fdc060a00) [pid = 15627] [serial = 402] [outer = (nil)]
[task 2018-06-16T01:54:11.299Z] 01:54:11     INFO - PID 15576 | ++DOMWINDOW == 41 (0x7f6fd4d6a800) [pid = 15627] [serial = 403] [outer = 0x7f6fdc060a00]

[taskcluster:error] Task timeout after 7200 seconds. Force killing container.
[taskcluster 2018-06-16 01:54:11.977Z] === Task Finished ===
[taskcluster 2018-06-16 01:54:11.978Z] Unsuccessful task run with exit code: -1 completed in 7202.314 seconds


The task seems to be running fine at the time of the timeout: It seems this chunk just needs more time, or fewer tests. Other linux ccov wpt chunks are long-running, typically 80 to 90 minutes.

Can we increase the number of chunks for linux ccov wpt? Or increase the max-run-time?

Comment 1

[Geoff Brown [:gbrown]]

I notice linux32/debug wpt2 is almost as bad: It fails frequently, but not every time. In non-failures, the run-time is  nearly 120 minutes.

https://treeherder.mozilla.org/logviewer.html#?job_id=183428732&repo=mozilla-central

Comment 2

[Geoff Brown [:gbrown]]

Also intermittent failures in linux64-qr/debug wpt2.

https://treeherder.mozilla.org/logviewer.html#?job_id=183412848&repo=mozilla-central


So maybe all of the linux/debug wpt tests need more chunks or a longer max-run-time.

Comment 3

[Geoff Brown [:gbrown]]

Let's try 18 chunks for linux*/debug:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1478b48c462fe248f10789e3301fc0a66825e61

Comment 4

[Geoff Brown [:gbrown]]

The longest running chunk - wpt8 in the try push - has a lot of long-running tests in /encoding/legacy-mb-tchinese/big5/big5-encode-href-errors-han.html, some of which (successfully) timeout. There were some recent changes to these in bug 1467143.

Comment 5

[Geoff Brown [:gbrown]]

Attachment: increase linux/debug wpt chunks from 12 to 18

The extra chunks give us a good buffer on linux*/debug, except for linux-ccov/debug, where wpt8 still takes 114 minutes (which might be good enough).

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8986172 [details] [diff] [review]
increase linux/debug wpt chunks from 12 to 18

Review of attachment 8986172 [details] [diff] [review]:
-----------------------------------------------------------------

that is a lot of chunks

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1586e4426384
Increase test chunks for linux/debug wpt; r=jmaher

Comment 8

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/1586e4426384

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/ac127e734ef3