1627085 - bugbug based scheduled used in |./mach try auto| should not run tests that are hidden behind --full

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Now that we are seeing some usage of |./mach try auto|, I am finding references to running many tests on android-hw. This is behind --full because of lack of scalable resources, but now it will get selected automatically if a given test is relevant to code changes and also runs on android-hw.

I see this going in one or another direction:

1. anything behind --full is not possible to schedule with |./mach try auto|
2. if something behind --full is scheduled to run, determine if it is scheduled to run on another config and if so, remove the config that is hidden behind --full. This allows us to schedule tests that are unique to a specific config and run those if the tests won't run elsewhere.

Comment 1

[Andrew Halberstadt [:ahal]]

Just adding a bit more context.. --full is a ./mach try fuzzy invention and doesn't have any meaning in the taskgraph. The list of tasks that are excluded is here:
https://searchfox.org/mozilla-central/source/tools/tryselect/selectors/fuzzy.py#34

We'll want to share that list in both places, which likely means moving it into the taskgraph somewhere.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

we could use other criteria, but --full seems to be the dividing line for what is not expected to be green or what has to be carefully selected.

Comment 3

[Andrew Halberstadt [:ahal]]

In mach try fuzzy --full just means select from literally every possible task. It includes all the release tasks like beetmover and balrog and stuff like that. The list of tasks you're thinking of is the one linked in comment 1.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, the fuzzy selectors list from comment 1, thanks for clarifying.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I assume we would add a filter here:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/optimize/bugbug.py#93

and reference:
https://searchfox.org/mozilla-central/source/tools/tryselect/tasks.py#32

since this is a decision task we should have the tasks.py in a local tree and can import it.

:ahal, is this what you think is the right path?

Comment 6

[Andrew Halberstadt [:ahal]]

Yeah, I think something like that would be fine. Though I would prefer if the list itself lived somewhere in taskgraph.util and then tryselect imports it from there. That way we don't add a dependency from taskgraph -> tryselect and avoid needing to add tryselect to the sparse profile (plus tryselect already depends on taskgraph).

I would also possibly try to implement this filter as a standalone optimization, and then add it to the composite strategies defined here:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/optimize/__init__.py#398

Though this could be a follow-up.

Comment 7

[Edwin Takahashi (:egao | infrequent contributor)]

If the task scheduling is handled from mach try auto then I should take this, because I'm making some core changes (again) to the task filtering in Bug 1630350.

Comment 8

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1627085 - apply mach try task blacklist filter to mach try auto

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c7a7b7056257
apply mach try task blacklist filter to mach try auto r=jmaher,marco

Comment 10

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/c7a7b7056257