1451805 - Define equivalent of "baseUrl" for exchanges

Status: RESOLVED FIXED

(Taskcluster :: Services, enhancement)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The documentation
  https://docs.taskcluster.net/reference/platform/taskcluster-queue/references/events
says "exchange/taskcluster-queue/v1/task-defined".  But that might not be the exchange name on another deployment.  It's not even clear if we would expect another deployment to follow the `exchange/<namespace>/` and `queue/<namespace>/` pattern.

More importantly, taskcluster-client needs to know how to generate the exchange names, and what queue names to use.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

related: https://github.com/taskcluster/taskcluster-docs/pull/246

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/taskcluster-docs

https://github.com/taskcluster/taskcluster-docs/commit/705284804e423dfe5e26ec3372089990eec1d108
Bug 1451805 - add docs of pulse naming conventions

https://github.com/taskcluster/taskcluster-docs/commit/6cae36a30cc1ed559f43be3803cf258f7ef80a9c
Merge pull request #246 from djmitche/bug1451805

Bug 1451805 - add docs of pulse naming conventions

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Proposal: fix the pulse namespace (as defined by tc-pulse), and then have tc-pulse provide an exchangePrefix (and queuePrefix) for each namespace, along with the existing username.

So for a client, given a rootUrl, wishing to listen to messages from tc-github:

 1. Configure tc-client with the rootUrl and call pulse.namespace(reference.name) ('github')
 2. Construct the exchange name as `<exchangePrefix>/<reference.version>/<exchange>`, e.g.,
    taskcluster-exchanges/github/v1/pull-request
    ---------------------------- -- ------------
 3. Listen on that exchange.

For a service, given its rootUrl, wishing to publish messages:

 1. Call pulse.claimNamespace(reference.name)
 2. Construct exchange names .. same from here on out

This would require a bit more configurability from tc-pulse:

 - mapping from (namespace, n) to a username (currently just `<username>-<n>`)
 - mapping from namespace to exchangePrefix
 - mapping from namespace to queuePrefix

That would need to be flexible enough to keep our existing exchange names, while also allowing multiple TC clusters to run on the same AMQP server (each will have a namespace `queue` that will need to map to different users, exchanges, etc.).

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Sorry, the info I need is: does that solution sound practical and does it address the issue?

Comment 5

[Brian Stack [:bstack]]

This seems really awesome.

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

> That would need to be flexible enough to keep our existing exchange names, while also allowing multiple TC clusters to run on the same AMQP server

Running multiple TC clusters on the same AMQP server seems like a bad idea. If so using virtual hosts would do the trick.

But this would effectively introduce a new exchangePrefix that tc-pulse have to distribute to all consumers.
hmm, I guess it could work. I would suggest a single TC cluster per AMQP vhost.

That way tc-pulse just returns a RabbitMQ connection string, and exchanges have the same names between deployments.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

In bug 1465166 I began building exchange names using projectName, which is fixed and not dependent on the rootUrl or anything like that.  This might make testing a little difficult, but we can worry about that later.

With that change, everything in exchanges.json is independent of rootUrl.