1377528 - Publish notification about completed try push to pulse

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder, enhancement)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Publish notification when all scheduled jobs for a try push are complete.

Our use case in Bug 1364564 is that we'll push a web-platform-tests job to try, then fetch+parse test-job logs when all jobs for that try push are complete.

So I'd like to listen for something like:
exchange/treeherder/v1/jobs-complete with routing key like try/<revision>

If the message body contained info about the resultset, that would be handy, too.

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Hey Cam, I chatted about this with Ed at the All-Hands, and he mentioned that it's something you might have an opinion about. Is there anything preventing Treeherder from notifying on a complete try push as described above? 

If it's a relatively small change and you can provide a bit of guidance, I can set aside some time to submit PR for it myself -- let me know.

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Would this get sent out only for jobs that use try syntax to initially schedule jobs, or would it also fire when all manually requested jobs (via Add New Jobs) complete? 

I guess it could fire any time all currently scheduled jobs complete (which would mean multiple notifications could be published for a single revision).

Comment 3

[Cameron Dawson [:camd]]

This sounds cool, yeah.  As Wes said, it's possible that you would get multiple "complete" notices for the same revision if all the in-progress jobs went complete, and then a new job was started.  But that seems reasonable.  That scenario would probably only happen with backfilling and retriggers, etc.

I'm guessing this would be a celery beat task that would run every minute or two and do a query to get all the ``Push``es where there are no jobs with result='unknown'.  Then publish the Pulse events to that exchange you mentioned.  The query would probably need to use ``aggregation`` with ``annotate``: https://docs.djangoproject.com/en/1.11/topics/db/aggregation/

Take a look at how the ``fetch-allthethings`` and ``pushlog`` celery beat tasks are done:
https://github.com/mozilla/treeherder/blob/690d35357d13b7781697cc37f2dc7051b5885e65/treeherder/etl/tasks/buildapi_tasks.py#L47-L47

And then check out how ``resultset_action``s are done:
https://github.com/mozilla/treeherder/blob/26b156af5ac9fc6a50ee9341cec478799d9b8910/treeherder/model/exchanges.py#L6-L6

You can most likely add this as another action in the TreeherderPublisher there.

That should get you going in the right direction.  But please let me know if you need further help.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I guess there could also be a period where all scheduled build jobs have completed, but none of the test jobs that depend on those build jobs have yet been scheduled.

Since almost everything is in Taskcluster land these days, maybe there's something we can query from the task graph for a push that says that every task in the graph has completed? This would still mean you'd get multiple notifications for jobs manually added later, but it might be better than polling all pushes every minute. Maybe taskcluster even already has a notification for this that you could listen to. :)

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

And they do: https://docs.taskcluster.net/reference/platform/taskcluster-queue/references/events#taskGroupResolved

Comment 6

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Thanks Cam, thanks Wes! 

I hadn't realized how much progress had been made with TaskCluster lately. As far as I can tell, all wpt-related jobs are run in TaskCluster these days.

Comment 7

[Ed Morley [:emorley]]

As part of a bug triage session earlier, we confirmed with Maja that using the Taskcluster pulse notifications was sufficient for this use case. Whilst Treeherder emitting these too might be useful for some other cases, it imposes a performance penalty and complexity during job ingestion that would be good to avoid if possible.