1282923 - define a nightly kind to taskgraph

Status: RESOLVED WONTFIX

(Firefox Build System :: Task Configuration, task)

Description

[Anthony Miyaguchi [:amiyaguchi]]

We should define a nightly kind in task-graph where we can configure the nightly graph. The kind defines the tasks to be run and the dependencies between them (and optionally how to optimize it). This kind will probably initially configured to depend on the legacy kind until the builds and tests are refactored into their own kinds.

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: bug 1282923: define a nightly kind in taskgraph

This introduces an entry point into the nightly graph. This is a skeleton for
further work. `mach taskgraph nightly` will kick off the nightly process with
given respository and revision based on a new nightly kind.  This will rope in
the legacy builds as dependencies for fennec, which will eventually be
transitioned into their own kinds (which I hope will make this look cleaner).

Review commit: https://reviewboard.mozilla.org/r/61698/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/61698/

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Comment on attachment 8766991 [details]
bug 1282923: define a nightly kind in taskgraph

This adds an entry point into the nightly graph at `mach taskgraph nightly`. It defines the nightly kind which will create the nightly graph and pull in the proper dependencies based on taskgraph.target_tasks:tasks_tasks_nightly. This also defines an empty kind which will eventually populate the templates for the various nightly tasks. 

This is a WIP, but provides a starting place to fill in the rest of the nightly graph. I have launched the nightly command successfully from within try that should start this process from tip. https://tools.taskcluster.net/task-inspector/#dj0f-8dDS2C8wg4Wk4MOAQ/

One area in particular that I would like feedback on in particular are the command line parameters for the mach command. We decided that having the user provide the parameters for the nightly build is preferable since we can specify an arbitrary revision to build. Below is how I call into the nightly command in the worker.

> repository='https://hg.mozilla.org/mozilla-central' && \
> rev=$(hg parent --template '{node}') && \
> ./mach taskgraph nightly --pushlog-id='131757' \
> --project='try' \
> --owner='[e-mail]' \
> --level='1' \
> --base-repository=${repository} \
> --head-repository='https://hg.mozilla.org/try/' \
> --head-ref=${rev} \
> --head-rev=${rev}

In order to use this with the hook service, I need to figure out whether I can start a new task without specifying a specific revision (ie GECKO_HEAD_REV: TIP). I also need to investigate proper scopes. If this task is run without the `--dry-run` option, it will fail because it does not have queue:* scopes. This is probably the next thing on my list of things to do, along with figuring out how dependencies will work.

Comment 3

[Jordan Lund (:jlund)]

Comment on attachment 8766991 [details]
bug 1282923: define a nightly kind in taskgraph

https://reviewboard.mozilla.org/r/61698/#review58630

I think creating nightlies through a decision task is a better approach than what you had before. Although, I'm not sure if your patch is just for testing or if we want to go with the approach of having a `./mach taskgragh nightly` permanently. that subcommand looks very similar to `./mach taskgraph decision` as it has the same arguments. We also have the issue of figuring out how to trigger a `nightly` (nightly graph) vs triggering a `decision` (CI graph) with the same gecko repository. By same gecko repo, I mean we can't mess with mach_commands.py or .taskcluster.yml as that will affect CI decision.

In my head, I was thinking we would still call `./mach taskgraph decision` but use a custom project, e.g. `mozilla-central-nightly` rather than the default, the repo name. In other words, we would be creating a decision task with a different `target` set to limit the graph to just nightly tasks.

I guess in order to do that, we would need a way to train mozilla-taskcluster to use a different `--project` if it was initiated by say a hook and we wanted it to be a nightly?[1] Not sure what the best way to do that or if that's the best approach.

https://github.com/taskcluster/mozilla-taskcluster/blob/master/src/jobs/taskcluster_graph.js#L114

::: taskcluster/mach_commands.py:190
(Diff revision 1)
> +                     help='email address of who owns this graph')
> +    @CommandArgument('--level',
> +                     required=True,
> +                     help='SCM level of this repository')
> +    @CommandArgument('--dry-run', action="store_true", help="dry run of this command")
> +    def taskgraph_decision(self, **options):

hm, so this overrides the existing taskgraph_decision method so that you can trick the current decision task into doing a nightly graph instead of CI?

Comment 4

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #3)
> Comment on attachment 8766991 [details]
> bug 1282923: define a nightly kind in taskgraph
> 
> https://reviewboard.mozilla.org/r/61698/#review58630

> I guess in order to do that, we would need a way to train
> mozilla-taskcluster to use a different `--project` if it was initiated by
> say a hook and we wanted it to be a nightly?[1] Not sure what the best way
> to do that or if that's the best approach.

let's leverage the experts in case my lack of knowledge in this area steers you off course.

dustin: given patch and comments, what are your thoughts?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I have two concerns with this direction.  One, as Jordan has identified, is that a new mach sub command is not necessary here -- what you want is a decision task that happens to create a nightly graph rather than one for a CI build, so `./mach taskgraph decision` is the right choice.  Related, that decision task needs some way to know that it should do different things for nightly than for CIs, even if all of the other parameters are equal.

The second issue is creating a new "nightly kind".  A kind defines a kind of task, and I'm pretty sure nightly task graphs aren't composed of "nighly tasks".  Rather, they are composed of build and test tasks just like a CI task graph, along with some different things.  It would be silly, I think, to redefine all of those build and test tasks for nightly builds, as then every time someone changes them for CI builds, that someone will forget to change them for nightly, the nightly build will fail, and we'll be sad.

The details depend on what "different things" means.  I imagine some combination of:

 1. different task details: mozharness/mach parameters, caches, chain-of-custody stuff, clobbering
 2. different platforms: maybe we don't build an ASAN nightly?
 3. fewer tasks: perhaps some of the test suites are omitted
 4. more tasks: additional l10n, funsize, balrog, etc. tasks

Taking those one at a time:

#1 can be addressed in the task generation code, based on a parameter -- something like parameters['triggered-by'] (values 'nightly' and 'push'), set in the decision task by `--triggered-by=nightly`.

  if parameters['triggered-by'] == 'nightly':
    task['payload']['command'].append('--nightly-run')

#2 and #3 can both be addressed in the target task set method, just selecting tasks on the basis of build platform or suite.

#4 would probably involve some new task kinds (named, maybe, 'l10n', 'funsize', 'balrog', etc.).  We have a choice in how we implement these.  If it's useful to have the full task graph for every job include those tasks, then we can always generate them, and omit them from the target task set on CI pushes.  However, I suspect that none of those tasks would work on try pushes (for security reasons), in which case it's just a waste of resources to carry around several thousand tasks in CI builds that will never be executed.  So the other option is to conditionalize generation of tasks by those task kinds on parameters['triggered-by'] == 'nightly'.

Finally, if my "silly" comment above is incorrect, there's the option of creating an entirely distinct task-graph configuration. This would be rooted in taskcluster/nightly, and have a structure parallel to taskcluster/ci.  Invoke `./mach taskgraph decision` with `--root=taskcluster/nightly` to use it.  I can see this working without having to duplicate the build/test jobs if one of the tasks generated in the nightly graph is, itself, a ci decision task.  Another task in the nightly graph would then depend on that ci decision task, download its task-graph.json, and create the remaining nightly tasks.  I don't think this is a good idea -- just bringing it up as a possibility.

Comment 6

[Jordan Lund (:jlund)]

anthony, shall we resolve this as WONTFIX or do you want to morph it to something else?

Comment 7

[Anthony Miyaguchi [:amiyaguchi]]

Yes, I don't think approach is needed any longer. There is no need to define a monolithic nightly kind ala the legacy kind. Bug 1277595 is more relevant to the overall discussion for nightlies now.