1443503 - groupResolved notifications are often sent with a considerable delay

Status: RESOLVED WORKSFORME

(Taskcluster :: Services, enhancement, P5)

Description

[Marco Castelluccio [:marco]]

I'm often getting groupResolved notifications the day after a build and all associated tests have finished.
The groupResolved notifications trigger a code coverage parsing task.
This means the coverage data is sometimes lagging behind.

Comment 1

[Marco Castelluccio [:marco]]

Not sure if bug 1387027 is related, but it might be.

Comment 2

[Brian Stack [:bstack]]

Are we sure that the late taskGroupResolved

Comment 3

[Brian Stack [:bstack]]

In irc today we talked about looking into taskGroup EEiuah8fQmO_vFfGZ5axgw. Indeed, the task group did resolve a day after most of the tasks finished (and I believe a day after anything you would see in treeherder finished). However, it does appear that the notification of taskGroupResolved which was received at 2018-03-06T18:15:40.982036+00:00 matches up with the time that the last task in the group was resolved which is BbmlqYN0SoWRe2vIIU7fPw. It resolved with deadline-exceeded at 2018-03-06T18:15:38.842Z.

This is most likely because a build that the build this test depended on failed and it does appear that X06XTQYHRR-BG0wyxv1K9w failed. It seems like taskGroupResolved is working pretty much as expected although it does also seem like it is not proving to be a useful tool for you given how it works. We use taskGroupResolved in taskcluster-github but also listen for failed tasks simultaneously so it fits our use-case there.

Could the right thing to do here to be to listen for taskFailed/taskException on any tasks in the group and use that knowledge to trigger coverage earlier when you know certain tasks won't end up running? You could also argue that we should change the semantics of the taskGroupResolved event but that might be a bit harder to do at this point.

:jonasfj, any thoughts on this?

Comment 5

[Brian Stack [:bstack]]

Maybe we ought to publish an event when all tasks are resolved other than those that depend on failed/exceptioned tasks?

Comment 6

[Brian Stack [:bstack]]

I feel like what we provide right now doesn't satisfy a simple/good use-case for task group events. Sorry to keep adding these in subsequent comments, I just keep thinking of more things to say :p

Comment 7

[Marco Castelluccio [:marco]]

The best option for me would be to have the additional event, which might be useful in general and not just for code coverage.

If that isn't feasible, I can implement what you do for taskcluster-github. It's a bit more complex as it requires to save state.

The other option for me is to define a task that depends on all the code coverage tasks (with dummy tasks to avoid the 100 dependency limit) and then wait for the task-completed notification for this task.

Comment 8

[Jonas Finnemann Jensen (:jonasfj)]

> The other option for me is to define a task that depends on all the code coverage tasks (with dummy tasks to avoid the 100 dependency limit) and then wait for the task-completed notification for this task.

In this case you need to set task.requires = 'all-resolved' (instead of the default), see docs:
  https://docs.taskcluster.net/reference/platform/taskcluster-queue/references/api#task

Yet, if some task that the build depends on fails, you still wouldn't get your task running.

@bstack, we could consider in the future that all dependent tasks should be resolved with exception: 'dependency-failed'
whenever a dependent task fails  and task.requires = 'all-completed' (as is default).
We can't do this anytime soon, as it would have impact on remaining users of `queue.rerunTask`.

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

It seems this bug hasn't seen activity in a while.

Marco, Brian, do you know if this is still happening and causing trouble?

Comment 10

[Marco Castelluccio [:marco]]

It's still happening, but since the delay is usually short it doesn't matter that much to me.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

In that case, let's tackle this as part of the move to postgres.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

We never really got to the bottom of any delay here -- the one case Brian looked at showed the notification being sent at the correct, if surprising, time.

For lack of data and lack of activity, after we've migrated everything to postgres, I'm going to close this as WORKSFORME.