1700774 - Organize the test transforms into smaller pieces

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P3)

Description

[Andrew Halberstadt [:ahal]]

In bug 1696041 I had to add a hack or two to allow the 'raptor' transforms to use by-variant before a by-app key. In general I think that splitting variants should be done as early as possible since it seems like a very common key to want to use handled_keyed_by with.

I'd propose to move them to an independent transforms file which loads before the tests.py transforms in general, and before the raptor.py transforms for the raptor case.

Comment 1

[Andrew Halberstadt [:ahal]]

This work was mostly done, but I'm going to repurpose this bug to generally re-organize the test transforms better.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Move test transforms to a 'transforms/test' directory, r?gbrown!

This includes:

transforms/tests.py -> transforms/test/__init__.py
transforms/raptor.py -> transforms/test/raptor.py

This is a pre-cursor to splitting the file up into multiple smaller files under
the new 'test' transform directory.

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Group 'raptor' specific test configs together, r?#perftest-reviewers!

In the test transforms we currently have general 'test_description_schema' and
a schema for raptor / browsertime tasks. Prior, the raptor schema was validated
before the test schema, and we were doing a lot of validation twice.

This revision switches the order such that the test schema is first evaluated,
and then the raptor one is. To accomplish this, we create a 'raptor' subconfig
that the 'test_description_schema' completely ignores (and leaves up to the
raptor schema to evaluate).

The benefit of this change is that all test tasks will have a single entry point
when they get into the 'test' transforms. This makes reasoning about their
configuration much easier and lays the ground work to start splitting even more
transforms out into other modules.

Depends on D132068

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Run test schema validation immediately after setting defaults, r?gbrown!

This ensures validation happens before we handle variants and raptor. It means
we no longer make any meaningful changes to the graph before validating.

Depends on D132069

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Convert 'raptor' transform into general purpose 'run_siblings' transform, r?gbrown!

We'd like to break down the test transforms into smaller well-ordered pieces.
This will allow us to run any transforms defined in files under the
'transforms.test' directory.

Depends on D132070

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Fix circular dependency on 'get_mobile_project' in variants transform, r?gbrown!

We have a 3 way circular dependency in the test transforms:

1. The 'raptor' transforms depend on 'split_variants'
2. The 'get_mobile_project' utility function depends on the 'raptor' transforms
3. The 'split_variants' transform depends on the 'get_mobile_project' utility function

The only reason things are working is that in step 3 we don't need the result
of 'get_mobile_project' to be perfect. This revision breaks the circular
dependency by checking for 'android' in the test-platform rather than relying
on 'get_mobile_project == geckoview'.

While this is admittedly not as precise, we only apply the particular variant
that needs this to geckoview tasks in the first place. In other words, this
revision does not cause any additional tasks to be added / removed.

Depends on D132071

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/197932d5aecb
[taskgraph] Move test transforms to a 'transforms/test' directory, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/0d0151af2ece
[taskgraph] Group 'raptor' specific test configs together, r=perftest-reviewers,sparky
https://hg.mozilla.org/integration/autoland/rev/05210e8d9e8d
[taskgraph] Run test schema validation immediately after setting defaults, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/4875751bb32d
[taskgraph] Convert 'raptor' transform into general purpose 'run_siblings' transform, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/60e50587018c
[taskgraph] Fix circular dependency on 'get_mobile_project' in variants transform, r=gbrown

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - Fix missed test import after renaming file, r?gbrown!

This reference was in a new test which wasn't on searchfox yet.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/92fe337bfdbe
Fix missed test import after renaming file, r=gbrown

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Split 'variant' transform to a new file, r?gbrown!

Many things need to be set "by variant", so therefore we need to resolve them
early on in the process. By pulling them out into a separate file that
explicitly runs before the "init.py" transforms, we A) make it harder
for someone to accidentally add their transform ahead of them, and B) reduce
some of the clutter in "init.py".

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Move almost all test transforms to an 'other.py' file, r?gbrown!

Future commits will tease this apart a bit more, but for now this helps
crystallize the order in which transforms are applied. The flow of the overall
test transforms goes something like this:

1. Enter 'transforms/test/init.py'
2. Validate all tasks against the test_description_schema
3. Run sibling transforms (starting with 'variant.py' and ending with 'main.py' for now)
4. Make the job description

As we pull more transforms out of 'main.py' and into their own smaller
transform files, it will be clear that the order in which these smaller files
run is important. Adding new transforms will no longer involve picking some
random spot to insert it.

Depends on D132408

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Move chunking related transforms to a separate file, r?gbrown!

Chunking transforms should always go last since there should never be
configuration that is different between chunks (other than what is set by the
chunking transforms themselves).

Running them last also ensures we don't do extra work as the number of tasks
grows by 10x or more after splitting chunks.

Depends on D132409

Comment 13

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700774 - [taskgraph] Move 'set_worker_type' transform to a separate file, r?gbrown!

This is being split to a new file mainly because the worker definitions take up
a lot of LOC. Getting this out of the main transforms makes it easier to
navigate.

Depends on D132410

Comment 14

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/197932d5aecb
https://hg.mozilla.org/mozilla-central/rev/0d0151af2ece
https://hg.mozilla.org/mozilla-central/rev/05210e8d9e8d
https://hg.mozilla.org/mozilla-central/rev/4875751bb32d
https://hg.mozilla.org/mozilla-central/rev/60e50587018c
https://hg.mozilla.org/mozilla-central/rev/92fe337bfdbe

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1646b431158b
Port bug 1700774 - [taskgraph] Move test transforms to a 'transforms/test' directory. rs=bustage-fix

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/922daf9d828a
[taskgraph] Split 'variant' transform to a new file, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/cf1a890796be
[taskgraph] Move almost all test transforms to an 'other.py' file, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/fefc7e4c82d6
[taskgraph] Move chunking related transforms to a separate file, r=gbrown
https://hg.mozilla.org/integration/autoland/rev/f5ba94a770d2
[taskgraph] Move 'set_worker_type' transform to a separate file, r=gbrown

Comment 17

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/922daf9d828a
https://hg.mozilla.org/mozilla-central/rev/cf1a890796be
https://hg.mozilla.org/mozilla-central/rev/fefc7e4c82d6
https://hg.mozilla.org/mozilla-central/rev/f5ba94a770d2