Open Bug 1534733 Opened 7 years ago Updated 7 months ago

Unsuccessful task run with exit code: NaN completed in xx.xx seconds

Categories

(Taskcluster :: Workers, task)

Product:
Taskcluster
Component:
Workers
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
REOPENED

People

(Reporter: aryx, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell infra])

+++ This bug was initially created as a clone of Bug #1504262 +++

https:/https://treeherder.mozilla.org/logviewer.html#?job_id=233431966&repo=autoland

[vcs 2019-03-12T18:04:48.922Z] clone [===============> ] 1063987271/2970548256 33s
[vcs 2019-03-12T18:04:49.922Z] clone [===============> ] 1087923424/2970548256 33s
[taskcluster 2019-03-12 18:04:57.905Z] === Task Finished ===
[taskcluster 2019-03-12 18:04:57.908Z] Artifact "public/logs" not found at "/builds/worker/logs/"
[taskcluster 2019-03-12 18:04:57.909Z] Artifact "public/build" not found at "/builds/worker/artifacts/"
[taskcluster 2019-03-12 18:04:58.322Z] Unsuccessful task run with exit code: NaN completed in 43.054 seconds

Component: Operations and Service Requests → Workers

Pete, any idea what causes teh NaN here?

Flags: needinfo?(pmoore)

Looks like a gecko-3-b-linux worker - Wander, any ideas?

Flags: needinfo?(pmoore) → needinfo?(wcosta)

(In reply to Pete Moore [:pmoore][:pete] from comment #3)

Looks like a
[gecko-3-b-linux](https://tools.taskcluster.net/groups/
PpvNb6sLTzCb0QA3aW4Nyg/tasks/BA7dThAxSpmpOyOKz6UFwQ/details) worker -
Wander, any ideas?

I will take a look at this when I have some time.

Flags: needinfo?(wcosta)

(closed as part of mass closure of old intermittent bugs)

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → INACTIVE

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=347380277&repo=autoland&lineNumber=841

Status: RESOLVED → REOPENED
Resolution: INACTIVE → ---
Summary: Unsuccessful task run with exit code: NaN completed in 43.054 seconds → Unsuccessful task run with exit code: NaN completed in xx.xx seconds

Many of these failures are on Android and affect various test suites. The test suite always stops to log at the same step (source):

[fetches 2024-06-05T21:26:49.832Z] Extracting /builds/worker/fetches/android-system-images-linux.tar.zst to /builds/worker/fetches
[fetches 2024-06-05T21:26:59.189Z] /builds/worker/fetches/android-emulator-linux.tar.zst extracted in 10.624s
[fetches 2024-06-05T21:26:59.189Z] Removing /builds/worker/fetches/android-emulator-linux.tar.zst
[taskcluster 2024-06-05 21:27:41.872Z] === Task Finished ===
[taskcluster 2024-06-05 21:27:41.874Z] Artifact "public/logs" not found at "/builds/worker/workspace/logs/"
[taskcluster 2024-06-05 21:27:41.875Z] Artifact "public/test" not found at "/builds/worker/artifacts/"
[taskcluster 2024-06-05 21:27:41.876Z] Artifact "public/test_info" not found at "/builds/worker/workspace/build/blobber_upload_dir/"
[taskcluster 2024-06-05 21:27:42.061Z] Unsuccessful task run with exit code: NaN completed in 59.165 seconds
Flags: needinfo?(aaron.train)

Pete, do you know where the worker sets the exit code?

Flags: needinfo?(aaron.train) → needinfo?(pmoore)

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #148)

Pete, do you know where the worker sets the exit code?

https://github.com/search?q=repo%3Ataskcluster%2Ftaskcluster%20%22run%20with%20exit%20code%3A%22&type=code

in turn my guess is that comes from here:

https://github.com/taskcluster/taskcluster/blob/195372c043acd26571de071b9ae68c32cbd8a100/workers/docker-worker/src/task.js#L956

I think this will be eventually resolved when the Docker Worker to Generic Worker migration is completed. We're not really actively maintaining Docker Worker for this reason, but if it is important that this is fixed, we can look into it, or would be happy to review a patch.

My guess is that a code path is followed inside async run() where this.exitCode never gets set.

Flags: needinfo?(pmoore)
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended] → [stockwell infra]

Hi Pete, there's been a recent spike in falures as you can see here, seems to be happening on Android and Linux, could you please take a look or assign this to someone? Thanks.

Flags: needinfo?(pmoore)

It is frequent and an annoyance but the failure frequency remained in its common range.

Flags: needinfo?(pmoore)

Note, this should be resolved by the decommission of Docker Worker, which is ongoing this year.

You need to log in before you can comment on or make changes to this bug.