Intermittent [taskcluster:error] Task appears to have failed intermittently - exit code 4 found in task payload.onExitStatus list
Categories
(Firefox Build System :: Task Configuration, defect)
Tracking
(Not tracked)
People
(Reporter: apavel, Unassigned)
References
Details
Failure log: https://tinyurl.com/yxz4dwmb
[task 2021-01-27T16:09:07.049Z] df -h
[task 2021-01-27T16:09:07.049Z] Filesystem Size Used Avail Use% Mounted on
[task 2021-01-27T16:09:07.049Z] overlay 457G 37G 397G 9% /
[task 2021-01-27T16:09:07.049Z] tmpfs 64M 0 64M 0% /dev
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /sys/fs/cgroup
[task 2021-01-27T16:09:07.049Z] shm 64M 0 64M 0% /dev/shm
[task 2021-01-27T16:09:07.049Z] /dev/mapper/and10--vg-root 457G 37G 397G 9% /test
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /proc/acpi
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /proc/scsi
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /sys/firmware
[task 2021-01-27T16:09:07.049Z]
[task 2021-01-27T16:09:07.049Z]
[task 2021-01-27T16:09:07.049Z]
[task 2021-01-27T16:09:07.049Z] script.py: exiting with exitcode 4.
[taskcluster 2021-01-27T16:09:07.923Z] Exit Code: 4
[taskcluster 2021-01-27T16:09:07.923Z] User Time: 6m46.441519s
[taskcluster 2021-01-27T16:09:07.923Z] Kernel Time: 1m34.559616s
[taskcluster 2021-01-27T16:09:07.923Z] Wall Time: 47m48.437351615s
[taskcluster 2021-01-27T16:09:07.923Z] Result: FAILED
[taskcluster 2021-01-27T16:09:07.924Z] === Task Finished ===
[taskcluster 2021-01-27T16:09:07.924Z] Task Duration: 47m48.439755655s
[taskcluster 2021-01-27T16:09:07.987Z] Uploading artifact public/logs/localconfig.json from file workspace/logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.194Z] Uploading artifact public/test_info/android-performance.log from file workspace/build/blobber_upload_dir/android-performance.log with content encoding "gzip", mime type "text/plain" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.333Z] Uploading artifact public/test_info/logcat-FA84C1A00221.log from file workspace/build/blobber_upload_dir/logcat-FA84C1A00221.log with content encoding "gzip", mime type "text/plain" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.630Z] Uploading artifact public/test_info/main_raw.log from file workspace/build/blobber_upload_dir/main_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.815Z] Uploading artifact public/test_info/resource-usage.json from file workspace/build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:09.202Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/XW5m4QVDTcS4OV4pk0yG7A/runs/5/artifacts/public%2Flogs%2Flive_backing.log with mime type "text/plain; charset=utf-8" and expiry 2022-01-27T09:43:34.813Z
[taskcluster:error] Task appears to have failed intermittently - exit code 4 found in task payload.onExitStatus list
@Richard did something change for android tests?
@Steve, can you also take a look? Any jit side changes that could explain this?
Thank you.
Comment 1•5 years ago
|
||
I'm not really the one to ask about jit side changes, but I'm not a bad one for what appears to be the real problem. This looks like a timeout exception (ADBTimeoutError) that is then handled incorrectly by the jit-test remote testing harness. The fix for the latter looks to be straightforward, and will hopefully allow this error to be classified better.
Relevant portion of the log:
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - Traceback (most recent call last):
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 869, in get_remote_results
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - yield run_test_remote(test, device, prefix, tempdir, options)
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 486, in run_test_remote
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - cmd, env=env, cwd=options.remote_test_root, timeout=int(options.timeout)
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/venv/lib/python3.6/site-packages/mozdevice/adb.py", line 2113, in shell_output
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL - raise ADBTimeoutError("%s" % adb_process)
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL - mozdevice.adb.ADBTimeoutError: args: adb wait-for-device shell export LD_LIBRARY_PATH=/data/local/tmp/test_root/bin&& cd /data/local/tmp/test_root/tests/tests && su -c '/data/local/tmp/test_root/bin/js -f /data/local/tmp/test_root/tests/tests/lib/prologue.js --no-ion --ion-warmup-threshold=10 --ion-offthread-compile=off --more-compartments --no-baseline --no-blinterp --fuzzing-safe --selfhosted-xdr-path /data/local/tmp/test_root/tmp/shell.xdr --selfhosted-xdr-mode decode -e '"'"'const platform="linux"'"'"' -e '"'"'const libdir="/data/local/tmp/test_root/tests/tests/lib/"'"'"' -e '"'"'const scriptdir="/data/local/tmp/test_root/tests/tests/tests/gc/"'"'"' --module-load-path /data/local/tmp/test_root/tests/tests/modules/ -f /data/local/tmp/test_root/tests/tests/tests/gc/bug-1565272.js', exitcode: None, stdout:
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - During handling of the above exception, another exception occurred:
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - Traceback (most recent call last):
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/jit-test/jit_test.py", line 587, in <module>
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - main(sys.argv[1:])
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/jit-test/jit_test.py", line 569, in main
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - ok = jittests.run_tests(job_list, job_count, prefix, options, remote=True)
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 814, in run_tests
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - ok = run_tests_remote(tests, num_tests, prefix, options, slog)
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 905, in run_tests_remote
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - ok = process_test_results(gen, num_tests, pb, options, slog)
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 745, in process_test_results
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - for i, res in enumerate(results):
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 874, in get_remote_results
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - sys.stderr.write("Error running remote tests: {}".format(e.message))
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL - AttributeError: 'ADBTimeoutError' object has no attribute 'message'
[task 2021-01-27T16:09:06.981Z] 16:04:01 ERROR - Return code: 1
[task 2021-01-27T16:09:06.981Z] 16:04:01 ERROR - No tests run or test summary not found
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - TinderboxPrint: jittest<br/><em class="testfail">T-FAIL</em>
[task 2021-01-27T16:09:06.981Z] 16:04:01 INFO - ##### jittest log ends
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL - # TBPL RETRY #
[task 2021-01-27T16:09:06.981Z] 16:04:01 WARNING - setting return code to 4
| Reporter | ||
Comment 2•5 years ago
|
||
Closing as a duplicate of bug 1500266.
| Comment hidden (Intermittent Failures Robot) |
Description
•