1613403 - central-as-beta, beta-as-release etc. simulations should not be optimized on schedules-component

Status: NEW

(Firefox Build System :: Task Configuration, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

jsreftests only get scheduled if relevant code changes. The first push which contains the changes will trigger the scheduling. This is problematic on Try if the build configuration changes but the code which triggers jsreftests is unchanged. When sheriffs do a central-as-beta simulation and a different developer has already pushed to Try based on the tip of central, the central-as-beta won't contain jsreftests.

This caused bug 1611686 to be detected with several days delay (another change which triggered the scheduler + weekend made the detection possible),

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Should this done by replacement rules in https://searchfox.org/mozilla-central/source/testing/mozharness/configs/merge_day/central_to_beta.py etc. (does it support regular expressions for multiline expressions)? A first run of ./mach taskgraph optimized with the tip of central contained 10 jsreftests (all on ccov) if I remember correct, but later runs of ./mach taskgraph optimized | grep "jsreftest" | wc -l all returned 81 lines. What is the command to reproduce the taskgraph without reftests based on https://hg.mozilla.org/mozilla-central/rev/2319a64a3cd8ef4ac0b5a3204da43f8c12bb48fc

Comment 2

[Tom Prince [:tomprince]]

You can simulate what the decision task of an particular revision would be with code update by passing -p task-id=<decision-task-id> or -p <url of parameters.yml> [1]. You can also pass a local file to see how changes would affect the taskgraph logic.

This should not be done via doing replacement rules, but rather by having mach try release set the appropriate parameters to be used by the decision task. I wonder if setting optimize_target_tasks to False in the parameters would end up running the right set of tests?

Another possibility would be to have some way to configure what set of files to consider changed for scheduling purposes. We currently only look at the changes in the push for scheduling, but test-verify looks at every commit not in m-u when determining the number of chunks to create. It might be useful to extend that logic to scheduling. That wouldn't help for this bug, but if it looked at commits not in m-b, that might address this issue.

[1] That allows one to use a URL like https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.revision.0016ade97e65a470bc97521ab8338210e74cdd02.taskgraph.decision/artifacts/public/parameters.yml to generate a URL based on a commit and tree. I'd be open to making -p more flexible and able to specify a tree+revision directly, if that is used often.