1737673 - Formalize the 'variant' subsystem in test transforms

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, enhancement)

Description

[Andrew Halberstadt [:ahal]]

Variants cover runtime configuration we set in tasks that tweak how Firefox behaves (typically prefs). We currently define them in a big JSON object in transforms/tests.py, but it's getting unwieldy to maintain and use.

In this bug I plan to:

1. Tests
2. Move definition to a yaml file
3. Create documentation (and checks to ensure it stays up to date)
4. Force an owner to be listed
5. Allow composing variants (e.g, variants: ["fission", "webgl-ipc", "fission+webgl-ipc"])
6. Possibly pull the logic to a new transform file?
7. ???

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - [taskgraph] Add a test for 'split_variants' in tests.py transforms, r?jmaher!

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Pull 'variant' definitions into a yaml file, r?jmaher!

This makes them easier to find and inspect.

Depends on D129826

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Use 'jsone' to replace variant filter functions, r?jmaher!

This ensures the filters stay alongside the variant definitions. I called the
key 'when' as 'filter' is ambiguous about whether the tasks that match are
being included or excluded.

Depends on D129827

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Validate variants against a schema, r?jmaher!

Ensures variants all conform to a well defined schema and don't
contain any extra or unexpected data.

Depends on D129828

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Add documentation for the 'variants' system, r?jmaher!

This also splits the blurb for the 'test' kind into a new file as it was
getting too long for kinds.rst. Having documentation for every kind in the same
file tends to lead towards uninformative documentation.

Depends on D129829

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Remove unused 'webrender' variant, r?jmaher!

Depends on D129830

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Stop setting 'run-on-projects' in variants.yml, r?jmaher!

This makes things less confusing as we no longer sneakily override the value
defined in the test configs.

Depends on D129831

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Remove duplicate 'webgl-ipc-profiling' variant, r?jmaher!

This is identical to the 'webgl-ipc' variant and only used in one place.

Depends on D129832

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Add a 'contact' key to variants in the yaml file, r?jmaher!,jgilbert!,cpeterson!

Listing contacts next to the variant will provide a point of contact in case
there are questions around cost / capacity or relative importance of a given
variant.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/98a659f60686
[taskgraph] Add a test for 'split_variants' in tests.py transforms, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/e38b571d248d
Pull 'variant' definitions into a yaml file, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/18eec84f404b
Use 'jsone' to replace variant filter functions, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/e1d2518592d4
Validate variants against a schema, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/7147d8490b9c
Add documentation for the 'variants' system, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/44bf02d34aaf
Remove unused 'webrender' variant, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/070efa174493
Stop setting 'run-on-projects' in variants.yml, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/6b374ffe6878
Remove duplicate 'webgl-ipc-profiling' variant, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/b7ad64578b84
Add a 'contact' key to variants in the yaml file, r=jmaher,jgilbert,cpeterson

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Support 'composite' variants, r?jmaher!

Often we have two variants that we want to test together. The only way to do
this currently is to create a 3rd variant that is the sum of the previous two.

This feature will allow tasks to define multiple variants using the '+' token.
E.g:

variants: ["fission", "webgl-ipc", "fission+webgl-ipc"]

The third variant there will first apply the 'fission' configuration, then the
'webgl-ipc' configuration (in order).

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1737673 - Remove 'webrender-sw-*' variants in favour of composite variants, r?jmaher!

This replaces the following variants with composites:
webrender-sw-a11y-checks
webrender-sw-fission
webrender-sw-wayland

The full task graph is identical after this change.

Depends on D130399

Comment 13

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/98a659f60686
https://hg.mozilla.org/mozilla-central/rev/e38b571d248d
https://hg.mozilla.org/mozilla-central/rev/18eec84f404b
https://hg.mozilla.org/mozilla-central/rev/e1d2518592d4
https://hg.mozilla.org/mozilla-central/rev/7147d8490b9c
https://hg.mozilla.org/mozilla-central/rev/44bf02d34aaf
https://hg.mozilla.org/mozilla-central/rev/070efa174493
https://hg.mozilla.org/mozilla-central/rev/6b374ffe6878
https://hg.mozilla.org/mozilla-central/rev/b7ad64578b84

Comment 14

[Ryan VanderMeulen [:RyanVM]]

(In reply to Cristian Tuns from comment #13)

https://hg.mozilla.org/mozilla-central/rev/98a659f60686
https://hg.mozilla.org/mozilla-central/rev/e38b571d248d
https://hg.mozilla.org/mozilla-central/rev/18eec84f404b
https://hg.mozilla.org/mozilla-central/rev/e1d2518592d4
https://hg.mozilla.org/mozilla-central/rev/7147d8490b9c
https://hg.mozilla.org/mozilla-central/rev/44bf02d34aaf
https://hg.mozilla.org/mozilla-central/rev/070efa174493
https://hg.mozilla.org/mozilla-central/rev/6b374ffe6878
https://hg.mozilla.org/mozilla-central/rev/b7ad64578b84

Backed out for breaking local Windows builds (bug 1739520).
https://hg.mozilla.org/mozilla-central/rev/47f2f2add0cae4cb58825876f635b39d4dfc4839

Comment 15

[Andrew Halberstadt [:ahal]]

Thanks for the backout, sorry about that!

Looks like it just needs to be joined to gecko_taskgraph.GECKO.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8e4b3a24e96a
[taskgraph] Add a test for 'split_variants' in tests.py transforms, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/de45d5de31dd
Pull 'variant' definitions into a yaml file, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/906cce4fe844
Use 'jsone' to replace variant filter functions, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/4e7fba3ff6ee
Validate variants against a schema, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/2c7aa54e0414
Add documentation for the 'variants' system, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/baedbe12cb81
Remove unused 'webrender' variant, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/9e7d1bac29a6
Stop setting 'run-on-projects' in variants.yml, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/11a908ecd5ae
Remove duplicate 'webgl-ipc-profiling' variant, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/c5d8a5c91ddd
Add a 'contact' key to variants in the yaml file, r=jmaher,jgilbert,cpeterson
https://hg.mozilla.org/integration/autoland/rev/829097878a30
Support 'composite' variants, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/3db46303ab27
Remove 'webrender-sw-*' variants in favour of composite variants, r=jmaher

Comment 17

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/8e4b3a24e96a
https://hg.mozilla.org/mozilla-central/rev/de45d5de31dd
https://hg.mozilla.org/mozilla-central/rev/906cce4fe844
https://hg.mozilla.org/mozilla-central/rev/4e7fba3ff6ee
https://hg.mozilla.org/mozilla-central/rev/2c7aa54e0414
https://hg.mozilla.org/mozilla-central/rev/baedbe12cb81
https://hg.mozilla.org/mozilla-central/rev/9e7d1bac29a6
https://hg.mozilla.org/mozilla-central/rev/11a908ecd5ae
https://hg.mozilla.org/mozilla-central/rev/c5d8a5c91ddd
https://hg.mozilla.org/mozilla-central/rev/829097878a30
https://hg.mozilla.org/mozilla-central/rev/3db46303ab27