1637583 - Consider using a client-provided list of relevant changesets for try pushes

Status: NEW

(Testing :: General, enhancement, P3)

Description

[James Graham [:jgraham]]

When we push things to try we get a list of files changed in the changesets and use that to decide which tests to run either from smart-scheduling or the explicit SCHEDULES rules. The implementation uses json-automationrelevance on hgmo, which boils down to the code at [1]. AFAICT this basically asks for a list of changesets in the current push and then, for try has a hack to also look at previous pushes since otherwise repushing the same revisions to try would show zero files changed and so not schedule any pushes.

In general there isn't a good correlation between the commits pushed to try and the files that were changed in the current patchset. In particular:

• The case handled by the current code where a branch is pushed to try with only a subset (or none) of the patches changing, but for which all the commits on the branch should contribute to the selected tests
• The case not handled by the current code where the first person to push new commits from autoland or central to try gets all the jobs scheduled for those commits as well.

Instead of relying on the server to figure out which files changed, it might make sense to figure out the base commit on the client and provide it in the try_task_config.json file. We already have the logic to do this for HG and Git in mozversioncontrol [2].

This approach has a number of nice properties:

• Automatically handles the problem of getting only commits on the current branch to the best of our abilities
• Allows the possibility of the user overriding the base if they e.g. only want to run tests for the topmost commit.

There are some issues to consider. For example mach try again should special case the base commit and recompute it for each push rather than using the cached value.

[1] https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hgext/hgmo/__init__.py#l510
[2] https://searchfox.org/mozilla-central/source/python/mozversioncontrol/mozversioncontrol/__init__.py#148

Comment 1

[Marco Castelluccio [:marco]]

(In reply to James Graham [:jgraham] from comment #0)

AFAICT this basically asks for a list of changesets in the current push and then, for try has a hack to also look at previous pushes since otherwise repushing the same revisions to try would show zero files changed and so not schedule any pushes.

Maybe this behavior wouldn't be so bad. E.g. if you scheduled Python tests as you changed some Python files, it doesn't make sense to run them again on a repush, unless you modified them again.

Comment 2

[James Graham [:jgraham]]

Maybe this behavior wouldn't be so bad. E.g. if you scheduled Python tests as you changed some Python files, it doesn't make sense to run them again on a repush, unless you modified them again.

That doesn't work e.g. if your first push failed due to infra issues.