Closed Bug 1689171 Opened 5 years ago Closed 5 years ago

Intermittent [taskcluster:error] Task appears to have failed intermittently - exit code 4 found in task payload.onExitStatus list

Categories

(Firefox Build System :: Task Configuration, defect)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1500266

People

(Reporter: apavel, Unassigned)

References

Details

Treeherder link: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=XW5m4QVDTcS4OV4pk0yG7A.5&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=42791e22621d1dab5fff576448eccb45f6b3ca6a

Failure log: https://tinyurl.com/yxz4dwmb

[task 2021-01-27T16:09:07.049Z] df -h
[task 2021-01-27T16:09:07.049Z] Filesystem Size Used Avail Use% Mounted on
[task 2021-01-27T16:09:07.049Z] overlay 457G 37G 397G 9% /
[task 2021-01-27T16:09:07.049Z] tmpfs 64M 0 64M 0% /dev
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /sys/fs/cgroup
[task 2021-01-27T16:09:07.049Z] shm 64M 0 64M 0% /dev/shm
[task 2021-01-27T16:09:07.049Z] /dev/mapper/and10--vg-root 457G 37G 397G 9% /test
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /proc/acpi
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /proc/scsi
[task 2021-01-27T16:09:07.049Z] tmpfs 16G 0 16G 0% /sys/firmware
[task 2021-01-27T16:09:07.049Z]
[task 2021-01-27T16:09:07.049Z]
[task 2021-01-27T16:09:07.049Z]
[task 2021-01-27T16:09:07.049Z] script.py: exiting with exitcode 4.
[taskcluster 2021-01-27T16:09:07.923Z] Exit Code: 4
[taskcluster 2021-01-27T16:09:07.923Z] User Time: 6m46.441519s
[taskcluster 2021-01-27T16:09:07.923Z] Kernel Time: 1m34.559616s
[taskcluster 2021-01-27T16:09:07.923Z] Wall Time: 47m48.437351615s
[taskcluster 2021-01-27T16:09:07.923Z] Result: FAILED
[taskcluster 2021-01-27T16:09:07.924Z] === Task Finished ===
[taskcluster 2021-01-27T16:09:07.924Z] Task Duration: 47m48.439755655s
[taskcluster 2021-01-27T16:09:07.987Z] Uploading artifact public/logs/localconfig.json from file workspace/logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.194Z] Uploading artifact public/test_info/android-performance.log from file workspace/build/blobber_upload_dir/android-performance.log with content encoding "gzip", mime type "text/plain" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.333Z] Uploading artifact public/test_info/logcat-FA84C1A00221.log from file workspace/build/blobber_upload_dir/logcat-FA84C1A00221.log with content encoding "gzip", mime type "text/plain" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.630Z] Uploading artifact public/test_info/main_raw.log from file workspace/build/blobber_upload_dir/main_raw.log with content encoding "gzip", mime type "text/plain" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:08.815Z] Uploading artifact public/test_info/resource-usage.json from file workspace/build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2022-01-27T09:43:34.813Z
[taskcluster 2021-01-27T16:09:09.202Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/XW5m4QVDTcS4OV4pk0yG7A/runs/5/artifacts/public%2Flogs%2Flive_backing.log with mime type "text/plain; charset=utf-8" and expiry 2022-01-27T09:43:34.813Z
[taskcluster:error] Task appears to have failed intermittently - exit code 4 found in task payload.onExitStatus list

@Richard did something change for android tests?
@Steve, can you also take a look? Any jit side changes that could explain this?
Thank you.

Flags: needinfo?(sphink)
Flags: needinfo?(rpappalardo)

I'm not really the one to ask about jit side changes, but I'm not a bad one for what appears to be the real problem. This looks like a timeout exception (ADBTimeoutError) that is then handled incorrectly by the jit-test remote testing harness. The fix for the latter looks to be straightforward, and will hopefully allow this error to be classified better.

Relevant portion of the log:

[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -  Traceback (most recent call last):
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 869, in get_remote_results
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      yield run_test_remote(test, device, prefix, tempdir, options)
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 486, in run_test_remote
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      cmd, env=env, cwd=options.remote_test_root, timeout=int(options.timeout)
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/venv/lib/python3.6/site-packages/mozdevice/adb.py", line 2113, in shell_output
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL -      raise ADBTimeoutError("%s" % adb_process)
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL -  mozdevice.adb.ADBTimeoutError: args: adb wait-for-device shell export LD_LIBRARY_PATH=/data/local/tmp/test_root/bin&& cd /data/local/tmp/test_root/tests/tests && su -c '/data/local/tmp/test_root/bin/js -f /data/local/tmp/test_root/tests/tests/lib/prologue.js --no-ion --ion-warmup-threshold=10 --ion-offthread-compile=off --more-compartments --no-baseline --no-blinterp --fuzzing-safe --selfhosted-xdr-path /data/local/tmp/test_root/tmp/shell.xdr --selfhosted-xdr-mode decode -e '"'"'const platform="linux"'"'"' -e '"'"'const libdir="/data/local/tmp/test_root/tests/tests/lib/"'"'"' -e '"'"'const scriptdir="/data/local/tmp/test_root/tests/tests/tests/gc/"'"'"' --module-load-path /data/local/tmp/test_root/tests/tests/modules/ -f /data/local/tmp/test_root/tests/tests/tests/gc/bug-1565272.js', exitcode: None, stdout:
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -  During handling of the above exception, another exception occurred:
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -  Traceback (most recent call last):
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/jit-test/jit_test.py", line 587, in <module>
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      main(sys.argv[1:])
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/jit-test/jit_test.py", line 569, in main
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      ok = jittests.run_tests(job_list, job_count, prefix, options, remote=True)
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 814, in run_tests
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      ok = run_tests_remote(tests, num_tests, prefix, options, slog)
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 905, in run_tests_remote
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      ok = process_test_results(gen, num_tests, pb, options, slog)
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 745, in process_test_results
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      for i, res in enumerate(results):
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -    File "/builds/task_161176087559323/workspace/build/tests/jit-test/tests/lib/jittests.py", line 874, in get_remote_results
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO -      sys.stderr.write("Error running remote tests: {}".format(e.message))
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL -  AttributeError: 'ADBTimeoutError' object has no attribute 'message'
[task 2021-01-27T16:09:06.981Z] 16:04:01    ERROR - Return code: 1
[task 2021-01-27T16:09:06.981Z] 16:04:01    ERROR - No tests run or test summary not found
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO - TinderboxPrint: jittest<br/><em class="testfail">T-FAIL</em>
[task 2021-01-27T16:09:06.981Z] 16:04:01     INFO - ##### jittest log ends
[task 2021-01-27T16:09:06.981Z] 16:04:01 CRITICAL - # TBPL RETRY #
[task 2021-01-27T16:09:06.981Z] 16:04:01  WARNING - setting return code to 4
Flags: needinfo?(sphink)
Flags: needinfo?(rpappalardo)

Closing as a duplicate of bug 1500266.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → DUPLICATE
See Also: → 1642620
You need to log in before you can comment on or make changes to this bug.