1539932 - Add ability to easily define variants to the 'test' kind

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P2)

Description

[Andrew Halberstadt [:ahal]]

Sometimes we want to duplicate all of the tasks we have and run them with a modified 'pref'. Normally we add a key to all of the tasks and use a transform to duplicate them.

This is clunky to add new variants, because you have to create a transform for each new variant, and in that transform you need to be careful not to accidentally dupe your variant with combinations of other variants. Ideally, we could define a datastructure to store all variant infomation and allow tasks to define a list of variants. We could then use a single transform to do all the splitting and adding new variants in the future becomes much much easier.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This is clunky, but also potentially very expensive. Making it too easy runs the risk of a combinatorial explosion in test load. But I admit making something hard is not a great way to keep costs down :)

Comment 2

[Andrew Halberstadt [:ahal]]

To clarify, the current way we service these requests is the one where we risk a combinatorial explosion. This proposal aims to remove that risk.

You are right that the proposal might encourage more variants. Though I'd argue it's better for this process to be clear and easy anyway. It's a lot more obvious to spot someone adding something to ALLOWED_TEST_VARIANTS (or whatever) than it is some random key to the task definitions which might not mean anything special. If we define this structure in its own file, we could even set up a herald rule to require some sort of super review.

Comment 3

[Andrew Halberstadt [:ahal]]

For context, I filed this in preparation for fission which will definitely be a large enough change to warrant duplicating the world (maybe it will give us more leeway to turn off/reduce other areas).

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1539932 - [ci] Handle serviceworker/socketprocess test variants more generically, r?jmaher

We are starting to spin off more and more "variants" of test suites. These are
usually just duplicates of our pre-existing tasks, except with an additional
pref set.

Currently there are two variants (serviceworker-e10s and socketprocess-e10s),
but a third will be added soon (fission). This change ensures we handle these
types of requests in a consistent and well defined manner. It also splits tasks
in a loop, so we don't accidentally risk combinatorial explosion.

Variants should typically be reserved for very large changes that will impact
the entire codebase (think e10s).

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/614d913a1657
[ci] Handle serviceworker/socketprocess test variants more generically, r=jmaher

Comment 6

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/614d913a1657

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/df2d531ed9fb
Port bug 1539932 - remove serviceworker-e10s from C-C's taskcluster/ci/test/tests.yml. rs=bustage-fix