1513715 - Support associating a Treeherder "repository" with multiple GitHub branches

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, enhancement)

Description

[Simon Sapin (:SimonSapin)]

In Servo’s CI, we have the ability to run Try with only a subset of the test suites. This works by having the Homu bot push to different branches (like try-linux, try-wpt, etc.)

In Treeherder, we’d like to have a single view that shows all of these Taskcluster tasks at the same time. This means a single "repository" in treeherder/model/fixtures/repository.json.

Currently, each GitHub entry in repository.json as configured for a single git branch. I’d like to add support for configuring multiple branches.

The "proper" relational-database-way to do this would be to replace the branch column/field by a new Branch table, that would have Repository foreign key. However this seems like a larger code change and database migration than I’d like to get into.

A more "hackish" but possibly much easier way would be to allow the branch column to contain multiple comma-separate branch names.

Concretely, this would involve changing this query:

https://github.com/mozilla/treeherder/blob/bf7b05389e6/treeherder/etl/push_loader.py#L24-L26

… to replace `branch=transformer.branch` with either:

    branch__regex="(^|,)%s($|,)" % transformer.branch

or:

    Q(branch=transformer.branch) | 
    Q(branch__startswith="%s," % transformer.branch) |
    Q(branch__endswith=",%s" % transformer.branch) |
    Q(branch__contains=",%s," % transformer.branch)

Relevant docs: https://docs.djangoproject.com/en/1.11/ref/models/querysets/#regex

Comment 1

[Simon Sapin (:SimonSapin)]

I discussed this with Cameron on IRC. Ed, what do you think?

Comment 2

[Ed Morley [:emorley]]

Yeah seems fine for now :-)

Comment 3

[Simon Sapin (:SimonSapin)]

For this specific use case, ideally I could even configure a pattern like "try-*" rather than a hard-coded list of branch names. However this gets really tricky: push_loader.py is not finding strings that match the pattern: it has one given string (branch name) and finding SQL rows that have a pattern that would match it. I don’t know if this can be expressed as a Django ORM query.

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4371

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/84d92b43e49864c5275dcd7ff482a449bd7b79f0
Bug 1513715 - Support associating a "repository" with multiple GitHub branches (#4371)

* Add a test for a multi-branch repository
* Disable servo-try-taskcluster