1815618 - Run every modified/added test when using 'mach try auto'

Status: NEW

(Developer Infrastructure :: Try, defect)

Description

[Botond Ballo [:botond]]

In https://treeherder.mozilla.org/jobs?repo=try&revision=dd62532e9c72a117bbee3f1d129c7d5988c96618, the patch adds gtests, but mach try auto did not schedule a gtest run on any platform.

Comment 1

[Botond Ballo [:botond]]

(Note: I added some manually with "Add new jobs", that's why they're there now.)

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:marco, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[Marco Castelluccio [:marco]]

We should do what I suggested in bug 1728260 comment 9. We can't always rely on test-verify (e.g. for gtest we can't).

Comment 4

[Botond Ballo [:botond]]

Is fixing this on the team's roadmap?

The utility of mach try auto remains very low to me due to this issue (recent example: this Try push), forcing me to choose between spending more of my time on selecting a targeted set of tests in mach try fuzzy, or spending more CI resources on a blanket Try push that runs a large subset of all tests.

Comment 5

[Marco Castelluccio [:marco]]

We could implement this either in the optimizer in m-c or in bugbug itself.
Is there an easy way to link a test file or a test support file to the manifest that contains it?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I am not aware of test support files, but I am aware of test files. in the test-info-all task on central, there are some artifacts, you want the test-info-all-tests.json artifact.

here is a json object in this type of format:

{
  "tests": {
    "bz product::component": [
        {"manifest": [path [, path2]], "test": "path"},

    ]
  }
}

this might be too cumbersome to work around, so if there are other ideas using an artifact but different format I am all ears.

Comment 7

[Andrew Halberstadt [:ahal]]

Yeah, we already parse all the manifests as part of task generation. All that info should be in the test objects that get returned:
https://searchfox.org/mozilla-central/source/taskcluster/gecko_taskgraph/util/chunking.py#235

I think we should add this feature there.

Comment 8

[Marco Castelluccio [:marco]]

Do we have easy access to the patches that are part of the push in the loader?

If we implement this feature in the loader, then we would have to "duplicate" the functionality in the optimizer too, right?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

there is a function get_changed_files which should be what you are looking for:
https://searchfox.org/mozilla-central/search?q=get_changed_files&path=taskcluster%2F&case=false&regexp=false

Comment 10

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #8)

If we implement this feature in the loader, then we would have to "duplicate" the functionality in the optimizer too, right?

Ahal, is this correct? If so, does it make more sense to support this on the bugbug side?

Comment 11

[Andrew Halberstadt [:ahal]]

I think that's partially correct.

We could probably implement in a way such that when we compute the manifests to keep in the task generation phase, we can pass a list of manifests that should be kept no matter what down to the optimization phase. Similar to how the DO_NOT_OPTIMIZE parameter works for tasks.

I think it makes sense to implement this on the Gecko side, conceptually at least. But I don't know how difficult this would be in bugbug, so not sure where it would be easier.

One benefit of doing it locally in the Decision task is that we could use this optimization even if we aren't using bugbug, though this probably isn't a huge benefit in practice.