1239419 - Coalescer needs a threshold setting

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jake Watkins [:dividehex]]

The coalescer app needs a threshold setting so we don't coalesce on all task queues greater than one.  This should be probably be dynamically set via the api as opposed to a ENV setting.  All logic should be contained to the coalescer service and not within the worker.  One idea is to return an list with only the calling taskid if queue size is below current threshold and return the entire actual taskid list if equal to or above the threshold

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I tend to think that the threshold should be a combination: queue length > N && oldest task age > M.  Then even if we dump 100 tasks in with the same key within a short time, if we have the capacity to start them all immediately, we can avoid coalescing.

Comment 2

[Jake Watkins [:dividehex]]

Pending the review and merge of https://github.com/mozilla/tc-coalesce/pull/12, this will be complete.