1532360 - taskcluster queue: provide queue-max-age or queue-last-consumed

Status: RESOLVED INACTIVE

(Taskcluster :: Services, enhancement)

Description

[Andrew Erickson [:aerickson]]

Originally reported at https://github.com/taskcluster/taskcluster/issues/340.

github issue description:

////
I maintain a cluster of workers. We recently had some downtime, but it wasn't easy to detect. The queues kept growing, but that can be normal if lots of pushes/merges are happening.

If a queue is non-zero and a task hasn't been consumed in X minutes, alert.
If a queue's max-age is over X minutes, alert.
////

I'm happy to work on this.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

If you are willing to allow a few hours' "slop", you could use an Azure table to hold this information. Each queue process would hold an in-memory map, like the one you've built here, but only update the last-claim time in claimWork if it's more than N hours ago, and when doing so also write to an Azure table containing (provisionerId, workerType, lastClaim). Then the lastClaimed API method would then query the table and return the most recent lastClaim value it finds. Then you'd have an expiration cron task that clears out old entries from the table. This is similar to how we count last use of clients, for example.

In fact, now that I think of it, we've already done this - the lastDateActive for a workerType is the last time (within 6 hours) that a worker called claimWork. We could entertain the idea of changing N=6 to, say N=1 if that would be more helpful to you.

(Sorry I didn't bring that up earlier -- I have a vague memory of looking for this when you first filed #340 and not finding it -- I don't know how I missed it!)

As we begin work on worker manager, we can think about having a bit better resolution on the status of specific workers, then aggregating that status at the worker-type level. WM will know when an instance starts and stops (and for static pools of hardware there will probably be some mechanism to indicate manually that a host is "up and running" or "down for maintenance"), and could potentially gather information from the queue as to when the worker calls claimWork (indicating the worker process is running) and when the worker calls reclaimTask (indicating the worker process is executing a task). This means that, within the loose definition of "time" in a distributed system, the WM could indicate, per instance, whether it is up or down, whether the worker is operational, and how many tasks it is executing.

For your purposes, some simple sum queries would be able to determine the total "up" capacity in a worker pool, total number of operational workers, and total number of tasks executing. It would be easy to poll that every few minutes and compare it to the queue length and alert if tasks executing == 0 and queue length != 0 for a few consecutive minutes.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Owlish is going to be working on the worker manager, and is a good person to talk to about these ideas.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

This is generally something we're considering a requirement for the updated worker-status tracking in worker-manager. That requires postgres support, so it's not coming too soon, but basically w-m would have the information about whether and when a worker has claimed a task. So there's nothing to do here right now, but this will be solved in that larger project.