1133602 - Write an official node.js library and a new, more extensible Python package

Status: RESOLVED WONTFIX

(Webtools :: Pulse, defect, P1)

Description

[Mark Côté [:mcote]]

From what I saw, TaskCluster's Python Pulse code is a lot more flexible than mozillapulse.  We had talked in Portland about splitting it out from TaskCluster and making it the new general-use Python package for interfacing with Pulse.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

We haven't got a decent python consumer client for pulse. I have a python publishing client (somewhat done). And we have node clients for both consuming and publishing spread in two libraries to maximize confusion :)

So let me sum up with some structure:

### Clients for Publishing Messages
python:
  https://gist.github.com/jonasfj/33e4e0da765b05a601fc
  (Also duplicated in treeherder)
node.js:
  https://github.com/taskcluster/taskcluster-base/blob/master/exchanges.js
  (See tests for examples)

Notes:
  The clients for publishing messages encourages a pattern where you declare your exchanges
  in a source file. Then at runtime instantiate a publisher object from those declarations,
  this object publishes messages and validates against declared schemas and routing key format.


### Clients for Consuming Messages
node.js:
  https://github.com/taskcluster/taskcluster-client/blob/master/lib/pulselistener.js
  (See docs: https://github.com/taskcluster/taskcluster-client/blob/master/README.md)

browser:
  https://github.com/taskcluster/taskcluster-client/blob/master/lib/weblistener.js
  (Uses sockjs, I want to port it to pure websockets, so not ready to declare it stable yet)

Notes:
  The listening client is structure with objects such as:
   - PulseConnection (owns the TCP connection)
     - close()
   - PulseListener (owns an AMQP channel, a single queue and holds bindings)
     - constructor(prefetch, connection, queueName, maxLengthOfQueue) // queueName optional
     - bind({exchange: "...", routingKey: "..."})
     - on('message', callback)
     - on('error', callback)
     - resume()
     - pause()
     - deleteQueue() // delete queue if named (unnamed implies auto-delete)
     - close()
  The listening client doesn't implement automatic reconnection yet, we really should do that :)


### Options
Refactor the consuming client to have API similar to the node.js client. I don't think anybody wants
hardcoded exchange names.
And include the publishing code from the gist, then call that the canonical python client?
(Maybe someday we can extend PulseGuardian to allow uploads of exchange references)

Probably for another bug, but we should probably split out the node.js code too.
Maybe even make an official node.js client for pulse. I know there is a few out-dated
ones on npm already.

---
@mcote,
Do you like the structure of the node.js consumer library?
And how do you feel about the declarative nature of the publishing interface?

IMO, these guard against bugs, but they are also very opinionated. If you're happy with these patterns in the official python client, then we should absolutely refactor/split them out.

Comment 2

[Mark Côté [:mcote]]

Setting myself a reminder to respond.

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

Btw, the pmoore has/is doing some work for a pulse client in go.
http://godoc.org/github.com/petemoore/pulse-go/pulse

His concepts are pretty much a long the line of what I suggested above.
We might be able to formalize common API methods more, but this is pretty good.

Comment 4

[Mark Côté [:mcote]]

This looks good to me!

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

The link to the node publisher is broken, did the repo get moved? And is it usable outside of Taskcluster? And does it have docs on how to use it?

Comment 6

[Mark Côté [:mcote]]

These would be questions for Jonas.

Comment 7

[Jonas Finnemann Jensen (:jonasfj)]

https://github.com/taskcluster/pulse-publisher/

I added a bit of usage docs too:
https://github.com/taskcluster/pulse-publisher/blob/master/README.md#usage
It's rather geared towards how build stuff in taskcluster.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

We are no longer in the business of creating pulse-specific libraries. Use a normal AMQP client..