1204891 - [tracker] support .taskcluster.yml in hg.mozilla.org repositories without mozilla-taskcluster

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Pete Moore [:pmoore][:pete]]

Like we have in taskcluster-github, but for hg repositories. Eventually then gecko would use this mechanism to trigger taskcluster jobs.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I'm frustrated enough with m-t to take a go at this.  No promises :)

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I think I'll call it 'taskcluster-mercurial'.  Also, half the work is already done:

https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/notifications.html

I'd like to make this compatible with taskcluster-github, in the sense that you can use the same .taskcluster.yml for a repo that's mirrored between github and hg.m.o.  That should be pretty easy, as all of the github config is buried under task.extra.github, so I'll put similar things under task.extra.mercurial.

.taskcluster.yml already has template variables defined for most of the values the decision task needs.  The remainder are `project`, `level`, and `message`.  I think the former can be defined in an in-tree array, and the latter can be determined locally from the hg history.  So I think I will modify the decision tasks to calculate these things in-tree, thereby eliminating the need for any per-tree configuration in taskcluster-mercurial.

One of the cool outcomes of this is that we could, in principle, allow builds of pushes to user repos, too.  We can decide whether that's allowed by assigning scopes via roles.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, duh, I can just implement this as a hook, given bug 1225243 :)

Comment 4

[Pete Moore [:pmoore][:pete]]

Hi Alex, are you still working on this? Thanks!

Comment 5

[Alexandra Silva (:alexandra_)]

h(In reply to Pete Moore [:pmoore][:pete] from comment #4)
> Hi Alex, are you still working on this? Thanks!

Hi Pete, I am still working on trigger a hook by pulse messages! This task is the next..

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I'll get back into coordinating this.  Hopefully YD will have the hooks functionality for this ready soon :)

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Here's the information provided in the JSON context by mozilla-taskcluster:

{
  // tasks_for (string) - always 'hg-push' for mozilla-taskcluster (this is to
  // distinguish from other tools that might read the same template, such as
  // taskcluster-github)
  tasks_for: 'hg-push',

  push: {
    // push.owner (string) - the hg user that made the push (not always an email)
    owner: '..',

    // push.revision (string) - long hg revision hash
    revision: '..',

    // push.comment (string) - commit comment from the last commit in the push
    comment: '..'

    // push.pushlog_id (number) - id of this push in the hg pushlog; see
    // http://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/pushlog.html#writing-agents-that-consume-pushlog-data
    pushlog_id: ..,

    // push.pushdate (number) - epoch timestamp when this push occurred
    pushdate: ..,
  },

  repository: {
    // repo.url (string) - repository root URL (begins with `https://`, ends with `/`)
    url: '..',

    // repo.project (string) - alias for this repository
    project: '..',

    // repo.level (number) - SCM level for this repo
    level: ..,
  },

  // now (string) - the current time, useful for reproducible $fromNow invocations
  now: '..',

  // as_slugid (function) - given a label, generate a slugid.  Multiple calls with the
  // same label in the same push will generate the same slugiid, but different slugids
  // in different pushes.  Use this to generate taskIds, etc.
  as_slugid: function(label) { .. },
}

and here's an example v2 message:

{
  "type": "changegroup.1",
  "data": {
    "repo_url": "https://hg.mozilla.org/try",
    "heads": ["eb6d9371407416e488d2b2783a5a79f8364330c8"],
    "pushlog_pushes": [{
      "time": 14609750810,
      "pushid": 120040,
      "user": "[e-mail]",
      "push_json_url": "https://hg.mozilla.org/try/json-pushes?version=2&startID=120039&endID=120040",
      "push_full_json_url": "https://hg.mozilla.org/try/json-pushes?version=2&full=1&startID=120039&endID=120040"
    }]
  }
}

so I think we need:

• tasks_for: "hg-push"
• push:
  • owner: payload.data.pushlog_pushes[0].user
  • revision: payload.data.heads[0]
  • comment: needs to be fetched
  • pushlog_id: payload.data.pushlog_pushes[0].pushid
  • pushdate: payload.data.pushlog_pushes[0].pushlog_id
• repository: defined in the hook, created by ci-admin
• now: provided by json-e
• as_slugid: hooks wants to produce one task with a taskid of its own devising, so this needs revision

We'll need bug 1522891 to allow not creating a task for other events than "changegroup.1". If we can adjust the decision task to fetch the comment on its own, and solve the as_slugid question, then all of the above can be done easily in json-e.

This is my ideal: hooks directly creates the decision task, rather than how we do cron tasks, where a "cron task" task creates a "cron job" task which finally runs ./mach taskgraph decision.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Tom, what are your thoughts on the above in light of rTASKGRAPH? Would it make more sense to chain together two tasks in order to support a more general .taskcluster.yml format?

Comment 11

[Tom Prince [:tomprince]]

We chatted about this in IRC. The plan we came up with is:

• the hooks service will receive pulse message and generate a task from it.
• the hook will directly generate a decision task directly
• to start with, ci-admin will populate the hook from .taskcluster.yml
• we talked about having a higher-level format for describing the tasks, but the details are up in the air.

A few details that we didn't discuss:

• We probably don't want the pulse message at the top-level, so we can have other things in the context, like the task id.
• I think ci-admin should wrap the .taskcluster.yml to restructure the input to match the current format (much like actions are here).

At least in the short-term, this will make changing the .taskcluster.yml harder, and not self-serve. I think this is a reasonable trade-off, to get to using pulse.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

The way I implemented this ends up not depending on bug 1491371, although I'd still love to see that landed!

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

I'll loop back to delete the mozilla-taskcluster app (which is already disabled) in a few days, but otherwise this is done.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

I just deleted the TC creds.

Bug 1538033: delete pulse creds

Bug 1538034: delete treeherder API key

I sent a backup of the config DB to brian and coop, and have a copy myself. Encrypted.

And then I deleted the app. Angels sang.