1252144 - support writing decision tasks

Status: REOPENED

(Firefox Build System :: Task Configuration, task)

Description

[[e-mail] (:bhearsum)]

Right now it's not possible to have dependencies between tasks in tc-gh. The taskids are created by tc-gh, and not referenceable from .taskcluster.yml. This greatly limits the types of things you can do, eg: I want to run tests on pushes to https://github.com/mozilla/balrog, and then create a new docker image if the tests pass. I'd like to do this with two different tasks to avoid needing a big docker image that can do both.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

In-tree, we use decision tasks for this purpose, with great effect.  I think this is an area where the expressiveness of a YAML file just can't be enough.

In general, we should provide a way to build decision tasks.  I have only the vaguest ideas how to do that, though!

I added this to https://wiki.mozilla.org/TaskCluster/Round_Tuit_Box

Comment 2

[[e-mail] (:bhearsum)]

I learned today that Normandy is using a decision task with tc-gh (https://github.com/mozilla/normandy/blob/master/.taskcluster.yml) - is this fixed?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

It's *possible*, but normandy had to write that decision task from the ground up.  The idea here is to offer some framework or quick-start or something to make it easier.

Note that we will be adding the ability to have tasks depend on other tasks once we support JSON-e in .taskcluster.yml (bug 1335915).  That will make some of the simpler graphs (such as "build task -> test task") implementable *without* a decision task.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

This is covered by an RFC (and not being actively worked on)
  https://github.com/taskcluster/taskcluster-rfcs/issues/51

Comment 5

[Tom Prince [:tomprince]]

I'm planning on extracting taskgraph from mozilla-central in a reusable form in 2018 Q4.

Comment 6

[Tom Prince [:tomprince]]

http://hg.mozilla.org/users/mozilla_hocat.ca/ci-taskgraph is ready for review. The last few commits exists for testing without configuring having a repository with proper permissions set up.

The following is probably a useful command in reviewing this.

git diff --no-index --diff-filter=AM ../mozilla-central/taskcluster/taskgraph/ src/taskgraph

Also, I've tried to make the commit history readable.

It generates https://tools.taskcluster.net/groups/Oe1nA7IiTEW6EBi1WGICrA

The intention is for this to live at https://hg.mozilla.org/ci/taskgraph

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

This looks great!

It looks like this includes:

• overall graph generation process
• parameters (but fewer)
• docker image builds
• multiple "projects" in the same repo
• verifications framework (but not all verifications)
• hg specificity (pushlog_id, at least)
• optimization
• some of the existing hierarchy of transforms (task description, job)
• treeherder support
  (wow, that's a lot out of the box!)

and excludes:

• actions
• try support
• some specific optimizations
• release-related stuff
• generic-worker (at least in task.py)

(all of that's fine, I'm just trying to get a handle on the scope)

How do you see users of this software interacting with it? "Vendoring" it into their repo? "Hard forking" it into their repo? Depending on it as a requirement?

What do you think are the next steps?

Comment 8

[Tom Prince [:tomprince]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #7)

This looks great!

It looks like this includes:

• overall graph generation process
• parameters (but fewer)
• docker image builds
• multiple "projects" in the same repo

Note that project is how taskgraph talks about branches (in the sense that m-c and m-b are branches).

• verifications framework (but not all verifications)
• hg specificity (pushlog_id, at least)
• optimization
• some of the existing hierarchy of transforms (task description, job)
• treeherder support
  (wow, that's a lot out of the box!)

and excludes:

• actions

This and cron are high on my list of follow ups.

• try support

I want to support some sort of try, but at least for the initial projects, just running all the tasks is reasonable. I'm not sure what kind of support is needed for this.

• some specific optimizations
• release-related stuff

I think the removed optimizations and release related stuff is gecko specific. I do want to figure out how to introduce customization to handle similar things, but I'll need some use cases.

• generic-worker (at least in task.py)

Given that there aren't generic windows workers, this is lower priority.

(all of that's fine, I'm just trying to get a handle on the scope)

How do you see users of this software interacting with it? "Vendoring" it into their repo? "Hard forking" it into their repo? Depending on it as a requirement?

That is what the decsion-user docker image for. It includes the taskgraph package in it. My thought is that people will pin to a particular version (I've got some thoughts on changing the index used to be based on the commit hash) of the image, and use it that way.

What do you think are the next steps?

I'd like to get a repo landed, and this code landed an working, and the corresponding code in ci-admin.

actions and cron support are high priority. After that, I have a trello board tracking improvements I have in mind (invite sent). I'll probably start looking at some of our mobile products, and see what is required to port them to taskgraph.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Awesome! It's probably good to get this "in use" in a wide array of contexts to help suss out any remaining specific bits. I can volunteer to try to use this for the TC monorepo (which is currently using some fancy json-e in .taskcluster.yml). But that's a github repo so might already require some tweaks.

Comment 10

[James Graham [:jgraham]]

We are very intested in adding a decision task for wpt on GitHub, so would be very happy to try this out once it's ready for GitHub-backed usage.

Comment 11

[Tom Prince [:tomprince]]

Attachment: Bug 1252144: Add taskgraph repository and scopes; r?dustin

Comment 12

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4519

Comment 13

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/4b345de7396c5c91bc4adfde83c52a5bde0de3d8
Bug 1252144 - Add taskgraph repositories (#4519)

Comment 14

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4528

Comment 15

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/71ea5be77ed028dc4b5145c90e978b43c492f23b
Bug 1252144 - Adjust ui to handle rename of repository group ci-admin -> ci. (#4528)

Comment 16

[Tom Prince [:tomprince]]

Attachment: Bug 1252144: Add trust-domain scoped caches, scheduer-id and hgfingerprint to taskgraph repositories; r?bstack

Comment 17

[[e-mail] (:bhearsum)]

For it's worth, my original problem (having tasks that depend on other tasks) is solveable with the tc-gh v1 schema via as_slugid().

Comment 18

[Simon Sapin (:SimonSapin)]

I tried out Checks in Servo, in hope to improve the situation of bug 1548781. We have .taskcluster.yml creating one "decision" task, which runs a Python script which in turns creates other tasks. Only the decision task shows up in Checks, and the check run is marked as successful as soon as the decision task ends even though some other tasks may go on to fail.

A bot relies on this reporting to automatically merge PRs that pass all tests, so Checks is not viable for Servo yet because of this bug.

Comment 19

[Simon Sapin (:SimonSapin)]

Oops sorry, the previous comment was meant for bug 1533235.