1584217 - Intermittent [taskcluster:error] Task timeout after 3600 seconds. Force killing container.

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: aiakab [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=268549901&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/MfOy4ZeMRQq7s6Nb23w5Zg/runs/1/artifacts/public/logs/live_backing.log

---

[task 2019-09-26T12:44:23.481Z] [3457/3798] /builds/worker/fetches/llvm-project/build/stage2/clang/bin/clang++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/llvm-dwp -I/builds/worker/fetches/llvm-project/llvm/tools/llvm-dwp -Iinclude -I/builds/worker/fetches/llvm-project/llvm/include -fPIC -Qunused-arguments -gcc-toolchain /builds/worker/fetches/llvm-project/build/stage1/clang -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fno-exceptions -fno-rtti -MD -MT tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/DWPError.cpp.o -MF tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/DWPError.cpp.o.d -o tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/DWPError.cpp.o -c /builds/worker/fetches/llvm-project/llvm/tools/llvm-dwp/DWPError.cpp
[task 2019-09-26T12:44:23.594Z] [3458/3798] /builds/worker/fetches/llvm-project/build/stage2/clang/bin/clang++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/llvm-diff -I/builds/worker/fetches/llvm-project/llvm/tools/llvm-diff -Iinclude -I/builds/worker/fetches/llvm-project/llvm/include -fPIC -Qunused-arguments -gcc-toolchain /builds/worker/fetches/llvm-project/build/stage1/clang -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -fno-exceptions -fno-rtti -MD -MT tools/llvm-diff/CMakeFiles/llvm-diff.dir/DiffConsumer.cpp.o -MF tools/llvm-diff/CMakeFiles/llvm-diff.dir/DiffConsumer.cpp.o.d -o tools/llvm-diff/CMakeFiles/llvm-diff.dir/DiffConsumer.cpp.o -c /builds/worker/fetches/llvm-project/llvm/tools/llvm-diff/DiffConsumer.cpp

[taskcluster:error] Task timeout after 3600 seconds. Force killing container.
[taskcluster 2019-09-26 12:44:24.314Z] === Task Finished ===
[taskcluster 2019-09-26 12:44:24.314Z] Unsuccessful task run with exit code: -1 completed in 3630.724 seconds

Comment 1

[Arthur Iakab [arthur_iakab]]

dmajor can you please take a look?

Comment 2

[(Away)]

Attachment: Increase the timeout for the clang-aarch64 toolchain

With clang-8 sources, this task was already pretty close to the limit with typical times of 54-58 minutes.

In clang-9, enough new code was added that we're seeing frequent intermittent timeouts. Even the few successful builds come in just under the wire at 60 minutes.

Note for sheriffs, if the clang-9 update gets backed out, this patch can stay, we'll need it anyway.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/abc8072a0b3b
Increase the timeout for the clang-aarch64 toolchain r=nalexander

Comment 4

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/abc8072a0b3b

Comment 5

[Intermittent Failures Robot]

6 failures in 4530 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6

Platform breakdown:

• toolchains: 5
• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1584217&startday=2019-09-23&endday=2019-09-29&tree=all