1263821 - Add generic builder task definition

Status: RESOLVED WONTFIX

(Taskcluster :: General, defect)

Description

[Steve Fink [:sfink] [:s:]]

I sometimes need to build various packages in the automation environment (specifically, with the centos 6 versions of all the libraries and things). I can take over an existing task by ripping its guts out and replacing them with a script to do my build, but some of this is boilerplate that could be done once and reused for different targets.

Comment 1

[Steve Fink [:sfink] [:s:]]

Attachment: Implement generic linux64 builder task on try

With this, I can drop a script build-foo.sh into testing/taskcluster/scripts/misc/ and then do a try push

  try: -b o -p linux64-generic --script=build-foo.sh

and it will run that script (which has a couple of environment variables set, and is expected to produce its output in $UPLOAD_DIR).

I will attach another patch that uses this to build gdb, but note that my starting point for this was seeing that ehsan has a script in the scripts/misc/ directory already, for building clang, and has instructions at https://wiki.mozilla.org/User:Ehsan/How_to_build-clang that are pretty hairy and brittle. In theory, you could do a try push with --script=build-clang-linux.sh and you would get exactly the same result as you would following his instructions.

In reality, I tried doing so and ended up with https://tools.taskcluster.net/task-inspector/#QMWzfbZSQ9mYK7uTB5dA8g/0 which looks like the script is applying a patch that fails to apply to the version downloaded or something? Anyway, that's a separate problem that would be debuggable via --interactive.

Comment 2

[Steve Fink [:sfink] [:s:]]

Attachment: Script for pulling down and building gdb, usable with linux64-generic task

It's much easier to see an example, so here's my gdb building script. You can see the result at https://treeherder.mozilla.org/#/jobs?repo=try&revision=222fbd274e204d8847410467a8fb8c3fdce4f262

I should note that the purpose of both this gdb builder and ehsan's clang builder (I assume?) is to create an artifact that will later be manually uploaded to tooltool.

Comment 3

[Steve Fink [:sfink] [:s:]]

Comment on attachment 8740278 [details] [diff] [review]
Implement generic linux64 builder task on try

Ugh. I keep forgetting that splitting patches breaks bzexport's auto-obsoleting functionality. Sorry for the churn.

Comment 4

[Greg Arndt [:garndt]]

Comment on attachment 8740278 [details] [diff] [review]
Implement generic linux64 builder task on try

Review of attachment 8740278 [details] [diff] [review]:
-----------------------------------------------------------------

As this is only enabled for try and doesn't use caches, I don't see a huge problem in doing this if you're going to have generic tasks like these run often on try pushes and the only thing different is the script they run.

I would like to get some feedback from dustin who is working on rewriting our in-tree scheduling to see if this fits into what he's doing or if we should go on a better path for accomplishing this.

Comment 5

[Greg Arndt [:garndt]]

Comment on attachment 8740280 [details] [diff] [review]
Script for pulling down and building gdb, usable with linux64-generic task

Review of attachment 8740280 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/scripts/misc/build-gdb-linux.sh
@@ +10,5 @@
> +: ${UPLOAD_DIR:=$HOME/artifacts}
> +
> +# Fetch our toolchain from tooltool
> +cd $WORK
> +wget $GDB_URL

We should catch this and output an error that treeherder could parse if any of these steps fail.

Similar to here:
https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/builder/build-linux.sh#32

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

For things that are part of the build, we will soon have support for requiring that thing and rebuilding it on demand.  Toolchains like clang fall into that category.  I think it makes sense to support those as explicit targets in try, since they are a component of the full task-graph.  We will probably support specifying such tasks with --target-task-set=toolchain-linux64-clang, where that identifier is the task label (unique within the task graph) of that task.  There's support in the framework to take the outputs of one task and feed them as input (URLs) to the next.

However, I think that the --script=.. syntax suggests that 'push to try' is a useful way to run arbitrary tasks in taskcluster.  In other words, I feel like this is a very long workaround for not wanting to call queue.createTask(..) with a task definition and specifying the desktop-build image.  For truly one-off activities, I think createTask is the right choice.  I realize that there's a little trouble with where to put the script while you're debugging it, but I think --interactive fills that niche nicely.

Until that task-graph stuff is ready (which will be soon -- I keep getting distracted by bugs adding new tasks to the existing syntax!!), it might be best to keep something like this in your local patch queue.  In other words, you're welcome to use it in try, but I would prefer that it not be in-tree where others might use it or even build on it.

Comment 7

[Greg Arndt [:garndt]]

What dustin said above makes a lot of sense so it might be better to just keep this as a one off path that can be used on try if needed, but not merged in.  What do you think?

Comment 8

[Greg Arndt [:garndt]]

Comment on attachment 8740278 [details] [diff] [review]
Implement generic linux64 builder task on try

I think for the purposes of this bug, we might not want to merge this in.  removing r? for now.

Comment 9

[Steve Fink [:sfink] [:s:]]

Sorry, I should have replied to this sooner.

Sure, what Dustin said makes sense, and I'd rather have him maintain the conceptual integrity of things.

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> For things that are part of the build, we will soon have support for
> requiring that thing and rebuilding it on demand.  Toolchains like clang
> fall into that category.  I think it makes sense to support those as
> explicit targets in try, since they are a component of the full task-graph. 
> We will probably support specifying such tasks with
> --target-task-set=toolchain-linux64-clang, where that identifier is the task
> label (unique within the task graph) of that task.  There's support in the
> framework to take the outputs of one task and feed them as input (URLs) to
> the next.

That definitely fits one of my needs. (I have a task that runs some gdb/JS integration tests, but it requires a newer gdb than is available for CentOS 6, so having a toolchain-like setup is perfect.)

The other need (ok, want) is much more of a one-off build, which fits into:

> However, I think that the --script=.. syntax suggests that 'push to try' is
> a useful way to run arbitrary tasks in taskcluster.  In other words, I feel
> like this is a very long workaround for not wanting to call
> queue.createTask(..) with a task definition and specifying the desktop-build
> image.  For truly one-off activities, I think createTask is the right
> choice.  I realize that there's a little trouble with where to put the
> script while you're debugging it, but I think --interactive fills that niche
> nicely.

I don't actually know how to do this, but it sounds like you'd need to create a full task definition? My goal was to eliminate all of the boilerplate (eg timestamps with a sufficient gap between them, etc.) by reusing existing mechanisms. Is there a simple way to invoke queue.createTask that removes the drudgery? Things I would prefer to not have to specify:

  created
  deadline
  provisionerId
  workerType (this one is kinda ok)
  retries
  expires
  payload.maxRunTime
  payload.env.GECKO_HEAD_REPOSITORY (though this is obviously script-dependent)
  metadata.* (well, some of it would be ok)

Perhaps I should just figure out how to write a script that calls queue.createTask and fills all of that in for the sort of thing I'm looking for. Unless such a thing already exists?

> Until that task-graph stuff is ready (which will be soon -- I keep getting
> distracted by bugs adding new tasks to the existing syntax!!), it might be
> best to keep something like this in your local patch queue.  In other words,
> you're welcome to use it in try, but I would prefer that it not be in-tree
> where others might use it or even build on it.

That's fine with me. Though it sounds like I'd be better off with a script that creates a task, with lots of opinionated defaults. Maybe I'll try writing that next time.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

I think |mach createTask| might make a lot of sense.