1541420 - Redesign the 'test' loader

Status: NEW

(Firefox Build System :: Task Configuration, task, P3)

Description

[Andrew Halberstadt [:ahal]]

Tasks in the test kind have a special loader:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/loader/test.py

It uses several input files to load tests:

A) Normal task .yml files (e.g mochitest.yml)
B) test-sets.yml
C) test-platforms.yml

In order to load tasks we do the following steps:

1. Load the task definitions from the A) input files using the normal transform loader.
2. Read test-sets.yml and group task labels into a bunch of sets (only tasks that have an entry in test-sets.yml will be loaded, there is possibility for missed tasks here).
3. Read test-platforms.yml which maps test sets -> platforms
4. Loop through task definitions from step 1 and apply build dependencies based on which platforms run a set containing that task.

There are a few problems with this approach:

• It is confusing to update. There are so many different places that need to be modified and it's not clear how they all work together.
• It is inflexible. Every time there is a new edge case (e.g we want to run a specific task on a specific platform) we need to create a new "test set" just for that task. It ends up getting messy.
• It prevents us from using transforms that change the label. All labels that are listed in test-sets.yml need to be finalized in the task inputs. This leads to situations like this where tasks that should be handled by a split transform instead need to be duplicated over and over, doubling the size of the config files.

I don't really see the benefit of this loader over the source-test approach of using transforms to set up the build dependencies. Especially now that job-defaults is a thing.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This design originated by looking at the kind of cases that needed support at the time (that is, the distribution of jobs in Buildbot). Things have likely changed quite a bit since then, so it makes sense to reconsider the design.

One difference from source-test is that a particular YAML clause for a test is duplicated for each test platform where it should run. We wanted to avoid the case where we generate every test for every platform and then select out the desired tests, since that would create a lot of tasks and many of them would be nonsensical (e.g., junit on OSX).

That generation could also be done with a "split" transform, too, but I'm not sure that will address the first two issues you've raised.

I found it helpful to write out a list of the changes requested and other flexibilities required (by looking at bugs and revision histories) and then imagine how those would be addressed by a few alternative approaches. That list is probably already in your head! And I'm sure it's very different now than a few years ago.

Comment 2

[Andrew Halberstadt [:ahal]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #1)

One difference from source-test is that a particular YAML clause for a test is duplicated for each test platform where it should run. We wanted to avoid the case where we generate every test for every platform and then select out the desired tests, since that would create a lot of tasks and many of them would be nonsensical (e.g., junit on OSX).

Thanks for the context, I hadn't considered this!

This bug falls under the category of mild annoyance, so likely won't be worked on anytime soon.

Comment 3

[Andrew Halberstadt [:ahal]]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #1)

One difference from source-test is that a particular YAML clause for a test is duplicated for each test platform where it should run. We wanted to avoid the case where we generate every test for every platform and then select out the desired tests, since that would create a lot of tasks and many of them would be nonsensical (e.g., junit on OSX).

Actually re-reading I think I misinterpreted and this isn't true. We use by-platform to split out source-tests across platforms. We could just do the same here.

I think to solve this I'd tend to just stick to the base transform loader, and allow optional use of test-sets.yml/test-platforms.yml (depending if that task manually specifies a platform or not) in a transform itself.