Closed Bug 1319583 Opened 3 years ago Closed 3 years ago

No taskcluster retry on Android INFRA-ERROR

Categories

(Testing :: General, defect)

Version 3
defect
Not set

Tracking

(firefox53 fixed)

RESOLVED FIXED
mozilla53
Tracking Status
firefox53 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

Details

Attachments

(1 file)

Even though "INFRA-ERROR" is included in the retry regex:

https://dxr.mozilla.org/mozilla-central/rev/0534254e9a40b4bade2577c631fe4cfa0b5db41d/testing/mozharness/mozharness/mozilla/testing/errors.py#101

such failures do not appear to trigger a retry.

https://queue.taskcluster.net/v1/task/MFeuQsHuS3i3fs-0DX5t-A/runs/0/artifacts/public/logs/live_backing.log

[task 2016-11-22T22:21:29.192215Z] 22:21:29    FATAL - INFRA-ERROR: Unable to start emulator after 5 attempts
[task 2016-11-22T22:21:29.192635Z] 22:21:29    FATAL - Running post_fatal callback...
[task 2016-11-22T22:21:29.193006Z] 22:21:29    FATAL - Exiting -1
...
[task 2016-11-22T22:21:29.757442Z] cleanup
[task 2016-11-22T22:21:29.757495Z] + cleanup
[task 2016-11-22T22:21:29.757511Z] + local rv=255
[task 2016-11-22T22:21:29.757528Z] + [[ -s /home/worker/.xsession-errors ]]
[task 2016-11-22T22:21:29.757974Z] + cp /home/worker/.xsession-errors /home/worker/artifacts/public/xsession-errors.log
[task 2016-11-22T22:21:29.764425Z] + cleanup_xvfb
[task 2016-11-22T22:21:29.764650Z] pidof Xvfb
[task 2016-11-22T22:21:29.764834Z] ++ pidof Xvfb
[task 2016-11-22T22:21:29.768987Z] + local xvfb_pid=33
[task 2016-11-22T22:21:29.769025Z] + local vnc=false
[task 2016-11-22T22:21:29.774378Z] + local interactive=false
[task 2016-11-22T22:21:29.774457Z] + '[' -n 33 ']'
[task 2016-11-22T22:21:29.775133Z] + [[ false == false ]]
[task 2016-11-22T22:21:29.775391Z] + [[ false == false ]]
[task 2016-11-22T22:21:29.775653Z] + kill 33
[task 2016-11-22T22:21:29.775688Z] + screen -XS xvfb quit
[task 2016-11-22T22:21:29.833340Z] No screen session found.
[task 2016-11-22T22:21:29.833753Z] + true
[task 2016-11-22T22:21:29.833806Z] + exit 255
[taskcluster 2016-11-22 22:21:31.563Z] === Task Finished ===
[taskcluster 2016-11-22 22:21:43.614Z] Unsuccessful task run with exit code: 255 completed in 1802.627 seconds
self.fatal() takes an optional exit_code parameter; if not specified (as here), it exits with the default, -1. Regardless of exit code specification, fatal() exits without any of the log parsing and exit code setting that would normally recognize "INFRA-ERROR" and trigger a TBPL_RETRY exit.
Summary: No taskcluster retry on Android x86 INFRA-ERROR → No taskcluster retry on Android INFRA-ERROR
Buildbot triggers retries based strictly on log parsing, so just logging "INFRA-ERROR" on buildbot successfully retries. 

Taskcluster triggers retries based strictly on exit code, which is set to -1 by default when exiting via fatal().

This patch ensures the exit code is set to TBPL_RETRY when an INFRA-ERROR causes an exit via fatal().


https://treeherder.mozilla.org/#/jobs?repo=try&revision=41288b9f8b41766e8adef8df19ff444c3de496f3 demonstrates retries with a forced INFRA-ERROR.
Attachment #8813922 - Flags: review?(kmoir)
Attachment #8813922 - Flags: review?(kmoir) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3e61ea3cb837
Trigger job retry on Android infra error; r=kmoir
https://hg.mozilla.org/mozilla-central/rev/3e61ea3cb837
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
You need to log in before you can comment on or make changes to this bug.