1335404 - Task 3 hours past deadline, yet still pending rather than deadline-exceeded

Status: RESOLVED WORKSFORME

(Taskcluster :: Services, defect)

Description

[Pete Moore [:pmoore][:pete]]

Task https://tools.taskcluster.net/task-inspector/#NcLFZyikQOeU6CySktaPKg/ exceeded its deadline 3 hours ago, but still has state "pending".

This could either be a delay or breakage to the task expiry process in the Queue. Hopefully just an abnormal delay.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

And, critically, it can't be cancelled in this state.  Perhaps the cancel operation should mark the task "deadline-exceeded" when it finds it to be past its deadline, rather than returning an error.

Comment 2

[Pete Moore [:pmoore][:pete]]

6 hours later now, still not resolved. Last queue deployment was 7 days ago: https://dashboard.heroku.com/apps/queue-taskcluster-net/activity

Comment 3

[Pete Moore [:pmoore][:pete]]

Jonas, any ideas?

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

Prolonged expiration is known to happen from time to time...

I'll search through logs. But it could be something gets dropped in case of an error that isn't handled right.
We've seen this happen a few time before, so maybe we have a bug.

We should probably scan all tasks to see how many of these are sitting around.
Afaik it's not a common issue.

Comment 5

[Pete Moore [:pmoore][:pete]]

It is resolved now - Jonas did you manually resolve, or did it get resolved by itself? If so, we can probably close this bug.

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

I didn't manually do anything... So maybe the message just got seriously delayed. Well, that's good news.

Thanks for checking up on this.