1880337 - stop scheduling variants that are not intended to run when using --full

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I have noticed many times on try that when using ./mach try fuzzy --full we end up scheduling variants on ALL possible permutations, when typically they are only intended to run on a much smaller subset.

We need to avoid this footgun, it confuses people every time. We also need to allow for people to run variants on new configs when they want. I think if they want to they should edit the taskcluster/ci/test/*.yml files.

Moving forward we should fix filter_tasks_by_paths: https://searchfox.org/mozilla-central/source/tools/tryselect/tasks.py#155.

I am wondering if there are other ./mach features we should be considering as well.

:ahal, do you have thoughts on this?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

ack, wrong bug for filter_tasks_by_path; not sure exactly how to fix this one- I will have to research. still interested if you have any thoughts on this :ahal.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I am thinking there is something we can do when the resulting task has run-on-projects: [] to filter those out.

Comment 3

[Andrew Halberstadt [:ahal]]

I think I disagree with the premise here.. --full running every possible task is working exactly as designed, and I'd be opposed to doing any sort of filtering based on run-on-projects, because that's what running ./mach try fuzzy without --full does :).

That said, I think the real problem you're trying to solve is that people don't understand what --full is and use it as a crutch when they can't find a task they're looking for. I can see a couple options to fix things:

1. Make it harder to use --full and strongly discourage folks from using it unless they know what they're doing.
2. Stop generating these permutations of tasks in the first place. But this likely means a re-write of the test loader and is solely a taskgraph change (no ./mach try changes involved)

The title as written makes me lean towards marking this as a WONTFIX. Some possible alternative titles might be:

• Only generate test tasks for permutations we actually want to run somewhere
• Make it more clear that ./mach try fuzzy --full includes invalid tasks

I think the first title in particular is the one you are looking for here.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

yes, I think "Only generate test tasks for permutations we actually want to run somewhere" is what we want- and as far as I know this is mostly an issue with variants. If we narrowed the scope to this, could end up with "Only generate test tasks for test variant permutations we actually want to run somewhere"?

I am not sure how to resolve that. I don't think the build and general test scheduling is a problem. Variants are sort of a hack

Comment 5

[Andrew Halberstadt [:ahal]]

I think you could use the when key in variants.yml, e.g:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/variants.yml#298

Though as you can see that would have the potential for long messy conditions. It should be possible to break those conditions into a multiline if statement though. Or perhaps we could make when support a list that gets implicitly OR'ed together.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1880337 - when using --full on try, do not schedule variants if not intended torun. r=ahal

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3257420fb1e2
when using --full on try, do not schedule variants if not intended torun. r=ahal,ci-and-tooling,aryx,taskgraph-reviewers

Comment 8

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/3257420fb1e2

Comment 9

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/3257420fb1e2