Closed Bug 1467668 Opened 6 years ago Closed 5 years ago

Intermittent Automation Error: mozprocess timed out after 2400 seconds running ['/usr/bin/python2.7', 'mach', '--log-no-times', 'build', '-v']

Categories

(Firefox Build System :: General, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Filed by: toros [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=182379160&repo=mozilla-central

https://queue.taskcluster.net/v1/task/WhDUbLStR0Cro6xRiUyFvw/runs/0/artifacts/public/logs/live_backing.log

[task 2018-06-07T21:20:21.522Z] 21:20:21     INFO -       Running `/builds/worker/workspace/build/src/rustc/bin/rustc --crate-name geckodriver testing/geckodriver/src/main.rs --crate-type bin --emit=dep-info,link -C opt-level=1 -C panic=abort -C debuginfo=2 -C debug-assertions=on -C metadata=a70b236c66502c2d -C extra-filename=-a70b236c66502c2d --out-dir /builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -C linker=/builds/worker/workspace/build/src/build/cargo-linker -L dependency=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps -L dependency=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./debug/deps --extern mozversion=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libmozversion-b03b4f31f4b71ef6.rlib --extern mozprofile=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libmozprofile-7583e1d77b9509c9.rlib --extern clap=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libclap-b45d2075ff87d5ca.rlib --extern rustc_serialize=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/librustc_serialize-6caa148efd4dc579.rlib --extern webdriver=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libwebdriver-b3b810f0be7f0994.rlib --extern lazy_static=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/liblazy_static-cd5be94cee915c5a.rlib --extern log=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/liblog-4351621f045246b3.rlib --extern uuid=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libuuid-20815162ceec7075.rlib --extern chrono=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libchrono-4cfc581ac69319c9.rlib --extern mozrunner=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libmozrunner-5e8da84312b7d74f.rlib --extern zip=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libzip-7753cc43ebaa4886.rlib --extern hyper=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libhyper-542dd6efe6929613.rlib --extern regex=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/deps/libregex-65871dbbafc57239.rlib -C opt-level=2 -C debuginfo=2 -L native=/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver/./x86_64-unknown-linux-gnu/debug/build/bzip2-sys-9451385680f77695/out`
[task 2018-06-07T21:20:21.522Z] 21:20:21     INFO -      Finished dev [optimized + debuginfo] target(s) in 187.78 secs
[task 2018-06-07T21:20:21.522Z] 21:20:21     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver'
[task 2018-06-07T21:20:21.522Z] 21:20:21     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver'
[task 2018-06-07T21:20:21.522Z] 21:20:21     INFO -  ../../config/nsinstall -R -m 755 'x86_64-unknown-linux-gnu/debug/geckodriver' '../../dist/bin'
[task 2018-06-07T21:20:21.522Z] 21:20:21     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/testing/geckodriver'
[task 2018-06-07T21:22:49.299Z] 21:22:49     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/media/webrtc/trunk/gtest'
[task 2018-06-07T21:22:49.299Z] 21:22:49     INFO -  /builds/worker/workspace/build/src/clang/bin/clang++ -B /builds/worker/workspace/build/src/gcc/bin -std=gnu++14 -o ../../../../dist/bin/webrtc-gtest -Qunused-arguments -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -O1 -fno-omit-frame-pointer  /builds/worker/workspace/build/src/obj-firefox/media/webrtc/trunk/gtest/webrtc-gtest.list   -lpthread -flto=thin -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,--build-id -fuse-ld=lld -rdynamic -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib      -ldl  -lrt -lm -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -Wl,--version-script,/builds/worker/workspace/build/src/build/unix/stdc++compat/hide_std.ld -lXrandr
[task 2018-06-07T21:22:49.299Z] 21:22:49     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/media/webrtc/trunk/gtest'
[task 2018-06-07T21:23:00.650Z] 21:23:00     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/media/webrtc/trunk/gtest'
[task 2018-06-07T21:23:00.651Z] 21:23:00     INFO -  /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.dumpsymbols /builds/worker/workspace/build/src/obj-firefox/dist/bin/webrtc-gtest /builds/worker/workspace/build/src/obj-firefox/media/webrtc/trunk/gtest/webrtc-gtest_syms.track
[task 2018-06-07T21:23:00.651Z] 21:23:00     INFO -  Beginning work for file: /builds/worker/workspace/build/src/obj-firefox/dist/bin/webrtc-gtest
[task 2018-06-07T21:23:00.652Z] 21:23:00     INFO -  Processing file: /builds/worker/workspace/build/src/obj-firefox/dist/bin/webrtc-gtest
[task 2018-06-07T21:23:00.652Z] 21:23:00     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/host/bin/dump_syms /builds/worker/workspace/build/src/obj-firefox/dist/bin/webrtc-gtest
[task 2018-06-07T21:23:00.653Z] 21:23:00     INFO -  /usr/bin/objcopy:/builds/worker/workspace/build/src/obj-firefox/dist/bin/stAhWuoI: cannot create debug link section `/builds/worker/workspace/build/src/obj-firefox/dist/bin/webrtc-gtest.dbg': Invalid operation
[task 2018-06-07T21:23:00.653Z] 21:23:00     INFO -  Finished processing /builds/worker/workspace/build/src/obj-firefox/dist/bin/webrtc-gtest in 10.19s
[task 2018-06-07T21:23:00.654Z] 21:23:00     INFO -  Running: /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python /builds/worker/workspace/build/src/toolkit/crashreporter/tools/symbolstore.py -c --vcs-info --install-manifest=/builds/worker/workspace/build/src/obj-firefox/_build_manifests/install/dist_include,/builds/worker/workspace/build/src/obj-firefox/dist/include -s /builds/worker/workspace/build/src /builds/worker/workspace/build/src/obj-firefox/dist/host/bin/dump_syms /builds/worker/workspace/build/src/obj-firefox/dist/crashreporter-symbols /builds/worker/workspace/build/src/obj-firefox/dist/bin/webrtc-gtest
[task 2018-06-07T21:23:00.654Z] 21:23:00     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/media/webrtc/trunk/gtest'
[task 2018-06-07T21:24:40.578Z] 21:24:40     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/js/src/gdb'
[task 2018-06-07T21:24:40.580Z] 21:24:40     INFO -  /builds/worker/workspace/build/src/clang/bin/clang++ -B /builds/worker/workspace/build/src/gcc/bin -std=gnu++14 -o ../../../dist/bin/gdb-tests -Qunused-arguments -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-noexcept-type -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -O1 -fno-omit-frame-pointer -Werror  /builds/worker/workspace/build/src/obj-firefox/js/src/gdb/gdb-tests.list   -lpthread -flto=thin -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,--build-id -fuse-ld=lld -rdynamic -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/lib   ../build/libjs_static.a  ../../../config/external/nspr/pr/libnspr4.so ../../../config/external/nspr/libc/libplc4.so ../../../config/external/nspr/ds/libplds4.so -lm -ldl  -lrt -lm -ldl -Wl,--version-script,/builds/worker/workspace/build/src/build/unix/stdc++compat/hide_std.ld
[task 2018-06-07T21:24:40.580Z] 21:24:40     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/js/src/gdb'
[task 2018-06-07T21:24:40.841Z] 21:24:40     INFO -  make[4]: Nothing to be done for 'syms'.
[task 2018-06-07T21:24:44.668Z] 21:24:44     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/js/src/shell'
[task 2018-06-07T21:24:44.670Z] 21:24:44     INFO -  /builds/worker/workspace/build/src/clang/bin/clang++ -B /builds/worker/workspace/build/src/gcc/bin -std=gnu++14 -o ../../../dist/bin/js -Qunused-arguments -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-noexcept-type -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -O1 -fno-omit-frame-pointer -Werror  /builds/worker/workspace/build/src/obj-firefox/js/src/shell/js.list   -lpthread -flto=thin -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,--build-id -fuse-ld=lld -rdynamic -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/lib   ../build/libjs_static.a  ../../../config/external/nspr/pr/libnspr4.so ../../../config/external/nspr/libc/libplc4.so ../../../config/external/nspr/ds/libplds4.so -lm -ldl  -lrt -lm -ldl -Wl,--version-script,/builds/worker/workspace/build/src/build/unix/stdc++compat/hide_std.ld
[task 2018-06-07T21:24:44.670Z] 21:24:44     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/js/src/shell'
[task 2018-06-07T21:24:44.918Z] 21:24:44     INFO -  make[4]: Nothing to be done for 'syms'.
[task 2018-06-07T21:25:04.857Z] 21:25:04     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/js/src/jsapi-tests'
[task 2018-06-07T21:25:04.859Z] 21:25:04     INFO -  /builds/worker/workspace/build/src/clang/bin/clang++ -B /builds/worker/workspace/build/src/gcc/bin -std=gnu++14 -o ../../../dist/bin/jsapi-tests -Qunused-arguments -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-noexcept-type -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fuse-ld=lld -flto=thin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -O1 -fno-omit-frame-pointer -Werror  /builds/worker/workspace/build/src/obj-firefox/js/src/jsapi-tests/jsapi-tests.list   -lpthread -flto=thin -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,--build-id -fuse-ld=lld -rdynamic -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/lib   ../build/libjs_static.a  ../../../config/external/nspr/pr/libnspr4.so ../../../config/external/nspr/libc/libplc4.so ../../../config/external/nspr/ds/libplds4.so -lm -ldl  -lrt -lm -ldl -Wl,--version-script,/builds/worker/workspace/build/src/build/unix/stdc++compat/hide_std.ld
[task 2018-06-07T21:25:04.859Z] 21:25:04     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/js/src/jsapi-tests'
[task 2018-06-07T21:25:05.055Z] 21:25:05     INFO -  make[4]: Nothing to be done for 'syms'.
[task 2018-06-07T22:05:05.372Z] 22:05:05     INFO - Automation Error: mozprocess timed out after 2400 seconds running ['/usr/bin/python2.7', 'mach', '--log-no-times', 'build', '-v']
[task 2018-06-07T22:05:05.404Z] 22:05:05    ERROR - timed out after 2400 seconds of no output
[task 2018-06-07T22:05:05.435Z] 22:05:05    ERROR - Return code: -15
[task 2018-06-07T22:05:05.455Z] 22:05:05  WARNING - setting return code to 2
[task 2018-06-07T22:05:05.468Z] 22:05:05    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-06-07T22:05:05.483Z] 22:05:05    FATAL - Running post_fatal callback...
[task 2018-06-07T22:05:05.492Z] 22:05:05    FATAL - Exiting -1
[task 2018-06-07T22:05:05.507Z] 22:05:05     INFO - [mozharness: 2018-06-07 22:05:05.503009Z] Finished build step (failed)
[task 2018-06-07T22:05:05.510Z] 22:05:05     INFO - Running post-run listener: _summarize
[task 2018-06-07T22:05:05.523Z] 22:05:05    ERROR - # TBPL FAILURE #
[task 2018-06-07T22:05:05.537Z] 22:05:05     INFO - [mozharness: 2018-06-07 22:05:05.532096Z] FxDesktopBuild summary:
[task 2018-06-07T22:05:05.545Z] 22:05:05    ERROR - # TBPL FAILURE #
[task 2018-06-07T22:05:05.549Z] 22:05:05     INFO - Running post-run listener: copy_logs_to_upload_dir
This bug has failed 39 times in the last 7 days, only on linux64 opt build type.

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=195518097&repo=mozilla-inbound

 Automation Error: mozprocess timed out after 2400 seconds running ['/usr/bin/python2.7', 'mach', '--log-no-times', 'build', '-v']
[task 2018-08-23T14:19:43.763Z] 14:19:43    ERROR - timed out after 2400 seconds of no output
[task 2018-08-23T14:19:43.763Z] 14:19:43    ERROR - Return code: -15
[task 2018-08-23T14:19:43.763Z] 14:19:43  WARNING - setting return code to 2
[task 2018-08-23T14:19:43.763Z] 14:19:43    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-08-23T14:19:43.763Z] 14:19:43    FATAL - Running post_fatal callback...
[task 2018-08-23T14:19:43.763Z] 14:19:43    FATAL - Exiting -1
[task 2018-08-23T14:19:43.764Z] 14:19:43     INFO - [mozharness: 2018-08-23 14:19:43.764046Z] Finished build step (failed)
[task 2018-08-23T14:19:43.764Z] 14:19:43     INFO - Running post-run listener: _summarize
[task 2018-08-23T14:19:43.764Z] 14:19:43    ERROR - # TBPL FAILURE #
[task 2018-08-23T14:19:43.764Z] 14:19:43     INFO - [mozharness: 2018-08-23 14:19:43.764362Z] FxDesktopBuild summary:
[task 2018-08-23T14:19:43.764Z] 14:19:43    ERROR - # TBPL FAILURE #

:catlee: Can you please take a look at this bug?
Flags: needinfo?(catlee)
Whiteboard: [stockwell needswork]
Looks like clang is hanging or taking too long. We could try bumping the timeout?
Flags: needinfo?(catlee)
Product: Release Engineering → Firefox Build System
QA Contact: catlee
There have been 34 failures in the last week on Linux32&Linux64, build types Debug, Lto and Opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=197412964&repo=autoland&lineNumber=36688
Whiteboard: [stockwell unknown] → [stockwell unknown][stockwell needswork]
There are 10 failures in the last 30 days. we'll continue to monitor.
Whiteboard: [stockwell unknown][stockwell needswork] → [stockwell unknown]
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → INCOMPLETE
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.