Allow longer deadlines
Categories
(Taskcluster :: General, enhancement)
Tracking
(Not tracked)
People
(Reporter: marco, Unassigned)
References
Details
I have a few very long running tasks, and some tasks which depend on them.
I could separate them in two hooks, the first running the long tasks and the second triggered when the long running tasks finish, but this makes things more complex.
Could we allow longer deadlines?
|
||
Comment 1•5 years ago
|
||
Longer than 5 days? How long do you need?
|
Reporter | |
Comment 2•5 years ago
|
||
I can't tell exactly, maybe 7 days would be enough.
These tasks use a cache, so they usually finish in a couple of minutes. In the rare (once a year?) cases where the caches are empty, they can take a long time.
|
|
||
Comment 3•5 years ago
|
||
I think "5" was a way of spelling "a finite time". The idea of deadline is that a task's status is immutable past its deadline, making it much easier to reason about, and no longer part of the "hot" set of tasks.
I think "10" probably fits that bill just as well, and nicely encapsulates "a week". Would that be enough?
|
|
Reporter | |
Comment 4•5 years ago
|
||
Yes, that should be enough for my needs, thanks!
|
|
Reporter | |
Comment 5•5 years ago
|
||
Could the maxRunTime be increased too, from the current maximum of 86400?
|
|
||
Comment 7•5 years ago
|
||
|
|
||
Comment 8•5 years ago
|
||
|
|
||
Updated•5 years ago
|
|
|
||
Comment 9•5 years ago
|
||
I just found out that Azure doesn't support deadlines longer than 7 days. So, those week-long deadlines are likely to fail (we add 10 minutes to the deadline, so just fromNow('1 week') would always be too long).
https://docs.microsoft.com/en-us/rest/api/storageservices/put-message - see messagettl.
https://sentry.prod.mozaws.net/operations/taskcluster-cloudops-stage/issues/7134894/ from my testing
I will revert https://github.com/taskcluster/taskcluster/pull/2218. That hasn't been deployed anywhere but staging yet, anyway. I'll call this a minor change since the functionality never actually worked.
Note that we should be able to re-land this once Postgres is in use (bug 1436478)!
|
|
||
Comment 10•5 years ago
|
||
|
|
||
Updated•5 years ago
|
|
||
Comment 11•5 years ago
|
||
Did this get released in https://github.com/taskcluster/taskcluster/releases/tag/v24.3.0 or is there more work to do here?
|
|
||
Comment 12•5 years ago
|
||
Yes:
▶ [minor] bug 1604175
The maximum "deadline" has been reverted to 5 days, after its change to 10 days in v24.1.3. Values over 7 days caused internal server errors anyway, because the Azure queue backend cannot handle delays greater than that value. Since this functionality never worked, the revert is considered minor.
|
||
Comment 13•5 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #12)
The maximum "deadline" has been reverted to 5 days, after its change to 10 days in v24.1.3. Values over 7 days caused internal server errors anyway, because the Azure queue backend cannot handle delays greater than that value. Since this functionality never worked, the revert is considered minor.
Presumably this is possible now that we're in postgres?
I feel like there should still be huge caveats about tasks having deadlines this long, but I also assume people know what they're doing.
|
|
||
Comment 14•5 years ago
|
||
Yes, we could bump it back up now. Marco, is that still useful for you? I'd rather leave well enough alone if the change wouldn't provide value, just to minimize moving pieces.
|
|
Reporter | |
Comment 15•5 years ago
|
||
I do still have a couple of tasks which can take a very long time to run, but I'm caching intermediate results so I can manually retrigger them if they exceed the deadline.
So, to answer your question, this would save me from some manual work, but this manual work is so rare that I consider this bug just a "nice to have".
|
|
||
Comment 16•4 years ago
|
||
Description
•