1278639 - maxRunTime seems not to affect task

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Pete Moore [:pmoore][:pete]]

I'm not sure why - the code is there... Must be a bug. I've added some logging, maybe it helps.

Code: https://github.com/taskcluster/generic-worker/blob/v2.0.0alpha44/main.go#L852 (ignore the comment above it, which it does not relate to).

Example task that outlives the maxRunTime: https://tools.taskcluster.net/task-inspector/#MZ7q7fJSQeat3IcbqMj79Q/0

Extra logging added:
https://github.com/taskcluster/generic-worker/commit/df0baebc3b23dc2f64fbd9afd1b10be8071dcd6f

This will get picked up in next release, haven't made one yet, as this is not extremely urgent at the moment. The queue will time out the task if it exceeds deadline, and provisioner will kill workers after 96 hours, so the worst thing that happens is some tasks take longer than they should....

Comment 1

[Pete Moore [:pmoore][:pete]]

I've included in release v2.0.0alpha45, which has been rolled out to worker types win2012r2 and ttaubert-win2012r2 - so we should get this extra debugging for new tasks for these worker types....

This won't solve anything, but should give more insight into a possible cause.

Comment 2

[Pete Moore [:pmoore][:pete]]

Basically, I'm thinking it might be that the status can't get updated for some reason, which was executing quietly without any warning to the logs.

Looking at logs, no output from the go routine that should time the task out.

Could also be a strange concurrency problem, related to variable scope in go routine, or Sleep sleeping longer than it should and not waking up, or something else entirely.

This specifically is the extra logging line that reports when an attempt to update a task status could not be performed because the starting state that it was in was not one of the allowed states to transition from:

https://github.com/taskcluster/generic-worker/commit/df0baebc3b23dc2f64fbd9afd1b10be8071dcd6f#diff-06e2fe4e6ec3e79a5f7abda94a1c45e1R188

Comment 3

[Pete Moore [:pmoore][:pete]]

This is now fixed.