Closed Bug 1509673 Opened 6 years ago Closed 4 years ago

Intermittent win 2012 [taskcluster:error] Task aborted - max run time exceeded

Categories

(Firefox Build System :: General, defect, P5)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1589796

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: in-triage, intermittent-failure) 

Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=213675962&repo=try

https://queue.taskcluster.net/v1/task/TYIS9EAxRR6cRJeEaqQ4dQ/runs/0/artifacts/public/logs/live_backing.log

15:36:08     INFO - z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testAttribute
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testAttributes
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testEmpty
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testForwardInterface
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testInterface
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testMethod
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testMethodParams
15:36:08     INFO - TEST-PASS | z:\build\build\src\xpcom\idl-parser\xpidl\runtests.py | TestParser.testOverloadedVirtual
[taskcluster:error] Aborting task...
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 3016 (child process of PID 5076) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4848 (child process of PID 5076) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 5076 (child process of PID 4844) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 1704 (child process of PID 1496) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4396 (child process of PID 4960) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4844 (child process of PID 3716) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4468 (child process of PID 4184) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 1524 (child process of PID 4184) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 2096 (child process of PID 4184) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] ERROR: The process with PID 1496 (child process of PID 860) could not be terminated.
[taskcluster 2018-11-24T15:36:09.682Z] Reason: There is no running instance of the task.

[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4116 (child process of PID 1260) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 3104 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4376 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4184 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 3716 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 1260 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4960 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 860 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 2080 (child process of PID 1480) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 1480 (child process of PID 1756) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 1756 (child process of PID 3180) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 1520 (child process of PID 4000) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 3180 (child process of PID 4000) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4000 (child process of PID 4628) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 4628 (child process of PID 3484) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 3452 (child process of PID 2960) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 3484 (child process of PID 2960) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] SUCCESS: The process with PID 2960 (child process of PID 3140) has been terminated.
[taskcluster 2018-11-24T15:36:09.682Z] 
[taskcluster:warn 2018-11-24T15:36:09.686Z] exit status 255
[taskcluster 2018-11-24T15:36:09.751Z] ERROR: The process "2968" not found.
[taskcluster 2018-11-24T15:36:09.751Z] 
[taskcluster:warn 2018-11-24T15:36:09.751Z] exit status 128
[taskcluster 2018-11-24T15:36:09.812Z] ERROR: The process "3948" not found.
[taskcluster 2018-11-24T15:36:09.812Z] 
[taskcluster:warn 2018-11-24T15:36:09.812Z] exit status 128
[taskcluster 2018-11-24T15:36:09.869Z] ERROR: The process "3680" not found.
[taskcluster 2018-11-24T15:36:09.869Z] 
[taskcluster:warn 2018-11-24T15:36:09.869Z] exit status 128
[taskcluster 2018-11-24T15:36:09.916Z] ERROR: The process "2920" not found.
[taskcluster 2018-11-24T15:36:09.916Z] 
[taskcluster:warn 2018-11-24T15:36:09.916Z] exit status 128
[taskcluster 2018-11-24T15:36:09.962Z] ERROR: The process "2700" not found.
[taskcluster 2018-11-24T15:36:09.962Z] 
[taskcluster:warn 2018-11-24T15:36:09.962Z] exit status 128
[taskcluster 2018-11-24T15:36:10.007Z] ERROR: The process "2960" not found.
[taskcluster 2018-11-24T15:36:10.007Z] 
[taskcluster:warn 2018-11-24T15:36:10.007Z] exit status 128
[taskcluster 2018-11-24T15:36:10.007Z] === Task Finished ===
[taskcluster 2018-11-24T15:36:10.007Z] Task Duration: 1h59m58.6398093s
[taskcluster 2018-11-24T15:36:10.491Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:11.096Z] Uploading artifact public/logs/log_critical.log from file logs\log_critical.log with content encoding "gzip", mime type "text/plain" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:11.463Z] Uploading artifact public/logs/log_error.log from file logs\log_error.log with content encoding "gzip", mime type "text/plain" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:11.870Z] Uploading artifact public/logs/log_fatal.log from file logs\log_fatal.log with content encoding "gzip", mime type "text/plain" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:12.211Z] Uploading artifact public/logs/log_info.log from file logs\log_info.log with content encoding "gzip", mime type "text/plain" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:13.164Z] Uploading artifact public/logs/log_raw.log from file logs\log_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:13.962Z] Uploading artifact public/logs/log_warning.log from file logs\log_warning.log with content encoding "gzip", mime type "text/plain" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:14.321Z] Uploading artifact public/build/buildhub.json from file public\build\buildhub.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:14.681Z] Uploading artifact public/build/host/bin/mar.exe from file public\build\host\bin\mar.exe with content encoding "gzip", mime type "application/x-msdownload" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:15.122Z] Uploading artifact public/build/host/bin/mbsdiff.exe from file public\build\host\bin\mbsdiff.exe with content encoding "gzip", mime type "application/x-msdownload" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:15.511Z] Uploading artifact public/build/install/sea/target.installer.exe from file public\build\install\sea\target.installer.exe with content encoding "gzip", mime type "application/x-msdownload" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:18.676Z] Uploading artifact public/build/mozharness.zip from file public\build\mozharness.zip with content encoding "", mime type "application/x-zip-compressed" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:19.415Z] Uploading artifact public/build/setup.exe from file public\build\setup.exe with content encoding "gzip", mime type "application/x-msdownload" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:19.829Z] Uploading artifact public/build/target.awsy.tests.tar.gz from file public\build\target.awsy.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:20.189Z] Uploading artifact public/build/target.checksums from file public\build\target.checksums with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:20.646Z] Uploading artifact public/build/target.common.tests.tar.gz from file public\build\target.common.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:21.634Z] Uploading artifact public/build/target.cppunittest.tests.tar.gz from file public\build\target.cppunittest.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:22.404Z] Uploading artifact public/build/target.crashreporter-symbols-full.zip from file public\build\target.crashreporter-symbols-full.zip with content encoding "", mime type "application/x-zip-compressed" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:30.873Z] Uploading artifact public/build/target.crashreporter-symbols.zip from file public\build\target.crashreporter-symbols.zip with content encoding "", mime type "application/x-zip-compressed" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:32.257Z] Uploading artifact public/build/target.generated-files.tar.gz from file public\build\target.generated-files.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:33.005Z] Uploading artifact public/build/target.json from file public\build\target.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:33.384Z] Uploading artifact public/build/target.jsshell.zip from file public\build\target.jsshell.zip with content encoding "", mime type "application/x-zip-compressed" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:34.288Z] Uploading artifact public/build/target.langpack.xpi from file public\build\target.langpack.xpi with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:34.757Z] Uploading artifact public/build/target.mochitest.tests.tar.gz from file public\build\target.mochitest.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:37.066Z] Uploading artifact public/build/target.mozinfo.json from file public\build\target.mozinfo.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:37.426Z] Uploading artifact public/build/target.raptor.tests.tar.gz from file public\build\target.raptor.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:38.285Z] Uploading artifact public/build/target.reftest.tests.tar.gz from file public\build\target.reftest.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:40.793Z] Uploading artifact public/build/target.talos.tests.tar.gz from file public\build\target.talos.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:42.183Z] Uploading artifact public/build/target.test_packages.json from file public\build\target.test_packages.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:42.524Z] Uploading artifact public/build/target.txt from file public\build\target.txt with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:42.868Z] Uploading artifact public/build/target.updater-dep.tests.tar.gz from file public\build\target.updater-dep.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:43.261Z] Uploading artifact public/build/target.web-platform.tests.tar.gz from file public\build\target.web-platform.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:46.634Z] Uploading artifact public/build/target.xpcshell.tests.tar.gz from file public\build\target.xpcshell.tests.tar.gz with content encoding "", mime type "application/x-gzip" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:47.640Z] Uploading artifact public/build/target.zip from file public\build\target.zip with content encoding "", mime type "application/x-zip-compressed" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:50.647Z] Uploading artifact public/build/target_info.txt from file public\build\target_info.txt with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:50.977Z] Uploading artifact public/build/toolchains.json from file public\build\toolchains.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:51.473Z] Uploading artifact public/logs/certified.log from file generic-worker\certified.log with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:54.996Z] Uploading artifact public/chainOfTrust.json.asc from file generic-worker\chainOfTrust.json.asc with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2018-12-22T13:19:25.493Z
[taskcluster 2018-11-24T15:36:55.631Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/c0YOgx3RTPmUyZCYVsG5rA/runs
See Also: → 1494841
Summary: Intermittent JIntermittent win 2012 [taskcluster:error] Task aborted - max run time exceeded → Intermittent win 2012 [taskcluster:error] Task aborted - max run time exceeded
From 
https://treeherder.mozilla.org/logviewer.html#?job_id=213675962&repo=try, it looks like this is a code error not a task timing out error.  It looks like there was a problem with task timing out properly after the build failed.

Over the last 7 days this bug has 62 failures. These happen on windows2012-aarch64, windows2012-64, windows2012-32.

Here is the latest failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231052566&repo=autoland&lineNumber=91029

Flags: needinfo?(kmoir)
Flags: needinfo?(kmoir)
Keywords: in-triage

This spike in timeouts has receded, we will re-evaluate if it occurs again if the max runtime needs to increase or the instance type needs to be changed.

Over the last 7 days there are 33 failures present on this bug. These happen on windows2012-aarch64, windows2012-64

Here is the most recent log example: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239105056&repo=mozilla-central&lineNumber=38867

Flags: needinfo?(kmoir)

Looking at the most recent failures these are tending to be on trees where opt builds are pgo (beta, esr), or other random task timeouts.

We should up the timeout for these cases, but I don't think it's urgent given the recent frequency, and our solution here will change once the shippable configuration rides the trains.

Flags: needinfo?(kmoir)
Flags: needinfo?(kmoir) → needinfo?(cmanchester)

Looking at those retriggers we're timing out consistently on certain instance types. We're doing pgo builds here with the opt build configuration. We should be able to bump the max runtime to what the pgo builds have (3 hours instead of 2) on this branch to avoid this.

Flags: needinfo?(cmanchester)

We did that increase for beta 68 yesterday: https://hg.mozilla.org/releases/mozilla-beta/rev/5bd868ce2fd08433b06a727c679ab7fd3918274d

Can the task configuration be modified with the current syntax to always use 3h on non-trunk branches?

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #38)

We did that increase for beta 68 yesterday: https://hg.mozilla.org/releases/mozilla-beta/rev/5bd868ce2fd08433b06a727c679ab7fd3918274d

Can the task configuration be modified with the current syntax to always use 3h on non-trunk branches?

There may be a way to achieve this with a transform in the taskgraph code, but I think once we move to shippable builds on this branch this problem may go away.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.