1243844 - Flexible in-tree task config

Status: RESOLVED DUPLICATE of bug 1247703

(Firefox Build System :: Task Configuration, task)

Description

[Brian Stack [:bstack]]

Currently tasks are defined in yaml files that have done what yaml files always do: multiplied and ballooned.

We should find a way that tasks can be defined that is still declarative as much as possible, but allows for the mostly repetitive task definitions to not just be a bunch of copy-pasting.

While we're at it, we should make the dependency definition part of the configuration simpler.

[Background: https://walac.github.io/in-tree-config/]

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Ideally we'd also
 - support release tasks / release-promotion
 - support tasks of various "types" (for example, image-building tasks, compiler-building tasks, linting, `make check`, ..)
 - support mozci and try-extender by making clear what tasks *might* have been submitted, given different try flags
 - define the tasks directly, via the taskcluster proxy, rather than relying on the docker-worker to do so

Some of the minor annoyances I've run into:
 - There are a lot of axes along which jobs vary.  Some are in the same tree (opt vs. debug, linux vs. mac, etc.).
   Some depend on the tree (try jobs are not cached or coalesced; build jobs are).  Some depend on tree level
   (caches).  Some depend on the "type" of push (nightly vs. dep vs. release).  Breaking out these individual dimensions
   using separate YAML files stinks, but so does special-casing them in the mach command.
 - YAML inheritance can only add things (keys, array elements) not remove them, further limiting the inheritance design

Finally, there's a fair bit of thoughts hereabouts on build configuration.  The two schools of thought are:

[DRY] write each piece of configuration once, and it is magically applied exactly where it should be and nowhere else
  pro: simple to make changes along existing lines (e.g,. add a new flag to all linux builds)
  pro: diffs are often one-liners
  con: difficult to understand what the result of a change might be; common unintended consequences
  con: difficult to make a change that doesn't follow existing lines (e.g., new flag for 32-bit mochitests only)

[WET]: write each final configuration file separately, with lots of repetition
  pro: simple to make arbitrary changes with a suitably powerful text editor
  con: huge, repetitive diffs
  pro: results of a change are always clear, and scope is limited to the changed files
  con: similar files tend to "drift" apart as people make changes to the files they're familiar with

Mozharness is very much of the WET philosophy.  Buildbot-configs are very much of the DRY philosophy (there's a huge python dictionary and a bunch of code to transplant bits of the dictionary around).

I think a hybrid is probably best, but it will be good to lay out the principals early on in the process and make them clear and part of the review process.

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

Ideally, in-tree logic should also have some way to make tasks dependent on what files where touched.

Comment 3

[Gregory Szorc [:gps]]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #2)
> Ideally, in-tree logic should also have some way to make tasks dependent on
> what files where touched.

There is file matching code in the mozpack Python module for this. It's how the "Files" context manager from moz.build files works. See https://gecko.readthedocs.org/en/latest/build/buildsystem/files-metadata.html

Comment 4

[Steve Fink [:sfink] [:s:]]

So far, the yaml files have felt remarkably manageable to me so far, but I've only worked with a few examples and I agree with all of your concerns. Come to think of it, I have scraped up against the single-hierarchy limitations a couple of time already.

I'm in the DRY camp myself, but I chafe at the description of the buildbot configs as DRY -- there is still *plenty* of repetition. They're a bit of the worst of both worlds, in that they have lots of crazy complex logic to clone pieces around, and yet they're still repetitive and redundant and you have to change several places to make a change.

In YAML, you can &name things and refer back to <<: *names later, but I don't know if we use that.

Given that we implement our own $inherits, we *could* add features to the merging.

task:
  payload:
    $override(command):
      - replacement
      - value
  $delete(extra): null

or maybe

task:
  payload:
    $override(command):
      - replacement
      - value
  $delete:
    - extra

or perhaps

$inherits:
  from: 'foo.yml'
  replace:
    - task.payload.command
  delete:
    - task.payload.extra

But that might just be pushing things towards more complexity.

Are we even using all of the current flexibility properly? I just tried generalizing a layer of the randomly-chosen opt_linux32 -> opt_linux32_clobber -> linux32_clobber -> firefox_base hierarchy by making an opt_clobber and using it in place of opt_linux32_clobber. It sort of mostly worked. The main change is pasted below, but it seems like *some* amount of cleanup is possible.

----

diff --git a/testing/taskcluster/tasks/builds/opt_linux32_clobber.yml b/testing/taskcluster/tasks/builds/opt_clobber.yml
copy from testing/taskcluster/tasks/builds/opt_linux32_clobber.yml
copy to testing/taskcluster/tasks/builds/opt_clobber.yml
--- a/testing/taskcluster/tasks/builds/opt_linux32_clobber.yml
+++ b/testing/taskcluster/tasks/builds/opt_clobber.yml
@@ -1,18 +1,16 @@
 $inherits:
-  from: 'tasks/builds/linux32_clobber.yml'
+  from: 'tasks/builds/{{build_name}}_clobber.yml'
   variables:
-    build_name: 'linux32'
     build_type: 'opt'
 task:
   metadata:
-      name: '[TC] Linux32 Opt'
-      description: 'Linux32 Opt'
+      name: '[TC] {{Build_Name}} Opt'
+      description: '{{Build_Name}} Opt'
 
-  workerType: opt-linux32
+  workerType: opt-{{build_name}}

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Good points all.

Indeed, we may end up keeping the YAML inheritance *support* within the kinds system, and even using some more advanced features like this.  I do not want to try to implement *everything* using YAML, though, as I think that it's not a suitably expressive language for the massive complexity we need to represent.

We had a few people independently dive in on this project, so I'm going to dupe this bug to the tracker for the overall project, which also includes a proposal that seems to have been accepted as consensus.