Closed Bug 1016117 Opened 8 years ago Closed 7 years ago

Have treeherder emit pulse messages when resultsets are created so TC tasks can be attached


(Tree Management :: Treeherder, enhancement, P4)



(Not tracked)



(Reporter: jlal, Assigned: jonasfj)




(1 file, 2 obsolete files)

The current taskcluster + try integration creates it's own resultset and jobs. This is fine if we only wanted to create tasks and resultsets from TC but that is not the case. For some time we will need to support the current buildbot flow and other tooling that will eventually report into treeherder.

A easy solution to this is to have treeherder simply emit a pulse message when a resultset is created.

Something like:

exchange: treeherder/v1/resultset
routingKey: created.$repoName

With the current body of the resultset.

This will allow us to easily attach jobs to resultsets from different sources without racing to create the resultset (with potentially different data) and provides a much easier source of information for new commits (from try, github, hg, etc.., etc...).
We can probably work around not having this but this will make standing up TC alongside everything else almost trivial and would help other projects as well I am sure.
Whiteboard: [treeherder]
Component: TaskCluster → Treeherder
Product: Testing → Tree Management
Summary: [treeherder] Have treeherder emit pulse messages when resultsets are created so TC tasks can be attached → Have treeherder emit pulse messages when resultsets are created so TC tasks can be attached
Whiteboard: [treeherder]
Severity: normal → enhancement
OS: Mac OS X → All
Priority: -- → P4
Hardware: x86 → All
Blocks: 1043858
I've hacked a proposal for this here:

It needs the following:
 A) Proper naming of exchange, routing key entries, documentation
    I have a very limited idea about what data I'm actually posting, so a well-defined
    formalization of what various names cover and what they reference would be nice.
    What is 'revision'? and 'revision_hash'? which is the result-set-id, is there such a thing?
    And if there is no result-set-id, how is result-set identified? Should we call the exchange
    something else? Assuming there is no such thing as a result-set-id.
 B) 'repository_id' should be resolved to something, what? how?
 C) Additional comments to JSON schema, perhaps additional formalization
 D) Reorganizing the code here, I just added somewhere, along with
    (maybe belongs in a utilities/helper folder, and exchanges maybe somewhere else)
 E) Configure namespace for publisher in, should be treeherder-local, treeherder-dev,
    treeherder-staging or treeherder for local, dev, stage and production setting.

It would be great if someone from treeherder team could take a look at (A) through (E), I'll happily answer an questions. But I have limited idea about how treeherder works and what the data model looks like. So most of the documentation I wrote is pure guess work :)

 i)  This uses the public pulse user, from my understanding PulseGuardian is far from working,
     and no multi-user support.
 ii) This doesn't use confirm-channels, it might be possible to do with python, but my experiments
     with pyamqp only resulted in SSL not working, but still no blocking on publish.

Future work:
 1) Publish JSON schemas,
 2) Publish JSON reference as automatically generated by
 3) Static web-site with auto-generated docs from reference and JSON schemas (I have code for this)
I've fixed (B) thanks to jeads.

I'm now using this for developing locally.
Note. (E) should be addressed before this merged, but it's just a configuration issue.
Attached patch treeherder-pulse.patch (obsolete) — Splinter Review
See PR:
Note that there are two commits in this PR, the first one installs additional dependencies, see attached patch for the interesting commit, or:

The other commit it the vendored stuff.

Basically, the only thing that IMO blocks this is that we need to configure the `namespace` value differently on production, staging, development and local deployments.

If you could help me get the `namespace` property defined in configuration, that would be great. It'll also needs a review, if so let me know if you have any comments, please let me know... I would like to get this landed soon, it's been blocking my deployment of taskcluster-try integration for a while now.

Note, this will need some extra reconfiguration, when pulse guardian goes operational with multi-user support. We should ask mcote about how to make publishers when that happens.
I suspect it just adding username/password to the configuration file and getting rid of the `namespace` property. This will probably just be a configuration issue, ie. no new code.
Attachment #8497009 - Flags: review?(mdoglio)
Attachment #8497009 - Flags: feedback?(mdoglio)
Sorry if it's taking so long to give you a feedback, I'm planning to do it within the end of the week.
Comment on attachment 8497009 [details] [diff] [review]

Review of attachment 8497009 [details] [diff] [review]:

It looks great, can you please make it pep8 compliant?

::: treeherder/model/
@@ +106,5 @@
> +schemas = load_schemas(schema_folder)
> +
> +publisher = TreeherderPublisher(
> +    'public', 'public', schemas,
> +    # TODO: Use treeherder-local, treeherder-dev, treeherder-staging for

There's a SITE_URL variable available in the settings file that you can use to derive an identifier for the environment in use.
This is tipically "" for local development, for dev, etc. See for detail on how to access the django settings file.

Alternatively, you can create a new property in treeherder/settings/,
give it a default value and overwrite it in your local settings file. You can then access it the same way as before via django.conf.settings
Attachment #8497009 - Flags: review?(mdoglio)
Attachment #8497009 - Flags: review+
Attachment #8497009 - Flags: feedback?(mdoglio)
Attachment #8497009 - Flags: feedback+
Comments should be addressed now.

kombu update have been moved to compiled.txt as requested by jeads on irc, and submitted in a separate PR:

Note, you'll need to more PR 238, before you merge PR 220.

@mdoglio (or jeads),
Is there anything else I can do to make this land?
Just getting it on staging will allow us to deploy our try hookup code (already implemented and tested against his patch).
Assignee: nobody → jopsen
Blocks: 1080265
Blocks: 1080268
Blocks: 1081612
Commits pushed to master at
Bug 1016117 - Added/updated vendor libs to support publishing to pulse
Bug 1016117 - update celery and kombu version in requirements
Bug 1016117 - update celery project layout and worker init script
Merge pull request #239 from mozilla/bug-1016117-vendor-updates

Bug 1016117 - Added/updated vendor libs to support publishing to pulse
Depends on: 1082801
Attached file Migration to use pulseguardian users (obsolete) —
This should be a easy thing to review...

You can look at the last commit as it is all the difference from what you reviewed before.
Basically, it just not using the namespace and instead of using public pulse user it uses a username/password loaded from config file.

Note, I've tested this locally and messages does arrive. So it works.
Attachment #8497009 - Attachment is obsolete: true
Attachment #8504983 - Flags: review?(mdoglio)
Comment on attachment 8504983 [details] [review]
Migration to use pulseguardian users

I added a few comments to the PR
Attachment #8504983 - Flags: review?(mdoglio) → review-
Addressed logging, import issues, employed try/finally and responded to the rest of you questions.

I tested this locally and it works for me. I see the messages on pulse...
Attachment #8504983 - Attachment is obsolete: true
Attachment #8509728 - Flags: review?(mdoglio)
Depends on: 1087913
Commit pushed to master at
Merge pull request #250 from jonasfj/migrate-to-pulse-guardian

Migrate to pulse guardian (bug 1016117)
Thanks for the great job Jonas! Sorry if it took so long to review it.
I'll put it on stage on Monday
Attachment #8509728 - Flags: review?(mdoglio) → review+
That's great, once on stage we can finally deploy out try integration bits.

Please make sure the staging exchange is listed here:

Otherwise it can be really hard to guess the exchange name which depends in the username you registered.
It's on stage now, :jonasfj can you confirm it's working?
Sorry, I missed your comment, yes, I can confirm that it's working...
First real-world use-case for pulse inspector:

Can we add you as "maintainer" for the exchanges on:
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.