1363577 - Tons of "Task exception" emails received ~24 hours after cancelling

Status: RESOLVED INCOMPLETE

(Taskcluster :: General, enhancement)

Description

[Gregory Szorc [:gps]]

I just received dozens of "Task exception" emails derived from a Try push where I wanted email notifications. The problem is I cancelled the Try push via Treeherder ~24 hours ago.

https://tools.taskcluster.net/task-inspector/#WMCmIC0xRCGXwT9_nXAC7A is one such task that was just emailed about. It belongs to task group https://tools.taskcluster.net/task-group-inspector/#/CrWqoZrqTqysWJuC-jPQXw. The Try push was https://treeherder.mozilla.org/#/jobs?repo=try&revision=62aa5b55ee857a32d38b1b89bd58a7a411c1d89a.

I cancelled the Try push midway through because I could see things were failing and didn't want to waste resources. I received tons of emails at cancellation time yesterday. However, I'm completely surprised that I received emails for those tasks ~24 hours later. I have no idea what state the tasks were in, but it certainly feels like something didn't abort the tasks cleanly. I hope they weren't lingering around on workers because that would be a massive waste of resources.

Comment 1

[Greg Arndt [:garndt]]

These tasks were not executed by a worker, so no worries about waste of resources there.

I see quite a few tasks that have been canceled within that task group both in the inspector, as well as in mozilla-taskcluster (service that cancels tasks on behalf of actions within TH via pulse messages) so cancellations were working during this time.

These events show up in the mozilla-taskcluster logs such as:
 Received cancel event for task TDFNDu1nRRqXJZuxmHVALg by [e-mail]


I do not see mozilla-taskcluster receiving an action for that task ID nor another task ID from that group I checked that was marked as deadline-exceeded (FAjXUBpJQ86ft0FhseGPfQ ).  Perhaps TH did not send cancellations for some task IDs.

Currently Treeherder does not contact taskcluster directly to cancel tasks (this will change at some point).  As far as I'm aware of how TH cancels jobs is either directly by cancelling a single job, or by using the "cancel all" on a push.  By using the "cancel all" it's not cancelling all tasks within a task group in taskcluster, but rather whatever tasks that TH knows about at that particular time.  If the "cancel all" feature was used for this push, I wonder if TH was not aware of some of these jobs at the time of cancellation and so left it out.

Comment 2

[Greg Arndt [:garndt]]

I was just recently looking at TH code, and I do not believe "cancel all" in TH actually goes through and cancels all tasks on a push.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

The exception emails were about deadline expiration for tasks that were not cancelled.  This should be addressed by the cancel-push action, which will land at the same time as we land retrigger-as-an-action.  In the interim I think we will just live with this.