1225243 - Support triggering tasks in response to pulse messages

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Users would specify a topic, then substitute parameters from the message body into the task using some kind of templating language.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Given the current lack of enthusiasm for scheduled hooks (there's only one defined), and the lack of bugs blocked on this one, I'm going to punt on this.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

whoops, that was not the bug I meant to reassign :)

Comment 3

[akshitadvlp]

Hello,
Is this bug open for solving?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Nope, Alex is hard at work on this one.

Comment 5

[akshitadvlp]

Ohkay:)

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

un-assigning to make it clear that this is not currently in progress

Comment 7

[Pete Moore [:pmoore][:pete]]

Hi akshitadvlp, is this something you are still interested in?

Thanks!

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Some of Alex's work-in-progress is in https://github.com/taskcluster/taskcluster-hooks/pull/74 -- that may be useful as reference, but is likely full of conflicts after more recent changes to hooks..

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Reminder to myself to merge https://github.com/taskcluster/taskcluster-hooks/pull/108 (but that doesn't finish this bug yet)

Comment 10

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/taskcluster-hooks

https://github.com/taskcluster/taskcluster-hooks/commit/df67c37270db58e1d1a2ff1fc233cd322b6ea039
Merge pull request #108 from ydidwania/pulse-hooks

 Bug 1225243 - Support triggering tasks in response to pulse messages

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/taskcluster-hooks

https://github.com/taskcluster/taskcluster-hooks/commit/d2a157fbf620038f0181c158af459a957af29bed
[2] Bug 1225243 - Support triggering tasks in response to pulse messages (#109)

* Add listeners component

* Added tc-lib-pulse and client setup

* export triggerHookCommon

* Add listener setup and createListener handler

*  reconcile the Hooks table with the new Queues table

* Add bindings to Hook row definition

* load listeners and add config fields

* Remove queues and stop consumers when hook is removed

* add the bindings schema

* createListener takes a single argument

* adds tests for the pulse hooks and triggering

* serialize calls to `reconcileConsumers`

* create a new function deleteQueue. Do not stop client when terminating `listeners` component

* reconcile listeners on start-up

* Update bindings in Queues Azure table. Move syncing bindings into a separate function.

* Test updating bindings of a hook updates them in the queue

* use a default value for bindings in updateHook

* synchronize reconciling consumers

* Remove async-decorators and cleanup.

* use `queues` for the local variable

* factor out remove/have listener into functions

* use already injected components to load Listeners in tests

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Woo, so that's landed, and seems to be running just fine!

I think the next step is to modify taskcluster-tools to allow editing the bindings.  Then we can try it out and see if we find any bugs the tests and review process have missed.

Comment 13

[Yashvardhan Didwania [:ydidwania]]

Yes, thanks for the reviews.
There is also the blacklisting of exchanges. 
I looked at the tc-tools HooksManager, I will try to do that as soon as possible.

Comment 14

[[github robot]]

Commit pushed to master at https://github.com/taskcluster/taskcluster-tools

https://github.com/taskcluster/taskcluster-tools/commit/d77bf40a0323b66bdbcdc296ebb2a651d46d3650
Bug 1225243 - Support triggering tasks in response to pulse messages (#595)

* display bindings in hook definition

* make bindings editable for hooks

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

There is also the blacklisting of exchanges.

I think that's all that's left here?

Comment 16

[Yashvardhan Didwania [:ydidwania]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #15)

There is also the blacklisting of exchanges.

I think that's all that's left here?

I think yes.
Can you try to create a few pulse hooks to test if everything is working?

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

That's how I uncovered the bugs I filed on Friday. It's close, though!

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I triggered https://taskcluster-web.netlify.com/tasks/FOW6vSB3RxeFNnm5m0xi4w with

{
  "payload": {
    "pushlog_pushes": [
      {
        "time": 1548796336,
        "push_full_json_url": "https://hg.mozilla.org/users/dmitchell_mozilla.com/puppet/json-pushes?version=2&full=1&startID=4&endID=5",
        "pushid": 5,
        "push_json_url": "https://hg.mozilla.org/users/dmitchell_mozilla.com/puppet/json-pushes?version=2&startID=4&endID=5",
        "user": "[e-mail]"
      }
    ],
    "heads": [
      "a1aa789c71ae16c1c2f6dedd3e836f21d0494a63"
    ],
    "repo_url": "https://hg.mozilla.org/users/dmitchell_mozilla.com/puppet"
  },
  "_meta": {
    "sent": "2019-01-29T21:12:18.606899",
    "routing_key": "users/dmitchell_mozilla.com/puppet",
    "serializer": "json",
    "exchange": "exchange/hgpushes/v1"
  }
}

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

So, this is done -- the remaining dependencies are just cleanup.

Hooray!

Comment 20

[Yashvardhan Didwania [:ydidwania]]

Dustin, did we do the blacklisting of exchanges?

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Good memory! I thought we had a bug for that. Now we definitely do!