1789668 - Properly support comm_base_ref and comm_base_rev arguments to mach taskgraph decision

Status: RESOLVED FIXED

(Thunderbird :: Build Config, task)

Description

[Rob Lemley [:rjl]]

D156011 from bug 1788113 was a quick hack to keep the build going.

Ideally, comm_taskgraph.parameters.get_decision_parameters() will set parameters["comm_base_rev"] based on what is provided in the context used to render .taskcluster.yml.

Currently, arguments --comm-head-rev and friends are defined in m-c:/taskclusrter/mach_commands.py in get_taskgraph_decision_parser(). They need to be handled ingecko_taskgraph.decision.get_decision_parameters() because options (from the commandline) are not passed to comm_taskgraph.parameters.get_decision_parameters().

Looking at how other projects like Android Components are using upstream taskgraph, in order to determine parameters they are reading environment variables rather than relying on having options available. That gets around the need to set up commandline arguments in m-c:/taskclusrter/mach_commands.py.

The plan for this bug is to grab COMM_BASE_REPOSITORY, COMM_BASE_REV, COMM_HEAD_REPOSITORY, COMM_HEAD_REF, and COMM_HEAD_REV from the environment in comm_taskgraph.parameters.get_decision_parameters() and remove code from gecko_taskgraph and m-c:/taskclusrter/mach_commands.py that supports --comm-* arguments to mach taskgraph decision.

Comment 1

[Rob Lemley [:rjl]]

Attachment: Bug 1789668 - Move decision task handling of comm repos into comm_taskgraph. r=dandarnell

This will allow the removal of "comm" specific arguments to "mach taskgraph
decision" that set the comm repository values. The patch takes a cue from
the Fenix project (https://github.com/mozilla-mobile/fenix) which sets taskgraoh
parameters that are not supported in upstream taskgraphfrom an environment
variable that is set in .taskcluster.yml.

Depends on D156011

Comment 2

[Rob Lemley [:rjl]]

Attachment: Bug 1789668 - Remove comm- specific argument processing from "mach taskgraph decision". r=#releng-reviewers

Depends on D155978 from bug 1784232.
There is a patch to comm-central (D157388) that will read these values from
environment variables that are set in .taskcluster.yml anyway. Credit goes to
the Fenix and Focus Android projects for the inspiration.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6c13b23fb757
Remove comm- specific argument processing from "mach taskgraph decision". r=releng-reviewers,gbrown

Comment 4

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/6c13b23fb757

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c2b04f9b371b
Move decision task handling of comm repos into comm_taskgraph. r=dandarnell

Comment 6

[Geoff Lankow (:darktrojan)]

I assume we're now done here.

Comment 7

[Rob Lemley [:rjl]]

Attachment: Bug 1789668 - Follow-up [1/2]: Move get_decision_parameters into decision.py r=#thunderbird-reviewers

Moves get_decision_parameters() and associated code into comm_taskgraph.decision
to better mirror how upstream projects structure their taskgraph code.

Comment 8

[Rob Lemley [:rjl]]

Attachment: Bug 1789668 - Follow-up [2/2]: Fix Daily build cron decision task. r=#thunderbird-reviewers

Daily cron decision task failed because get_existing_tasks() was called in
gecko_taskgraph.decison.get_decision_parameters() prior to comm_head_rev and
comm_head_repository being added to parameters. The code that reads the associated
environment variables and sets those parameters runs towards the end of
gecko_taskgraph.decison.get_decision_parameters().

To work around that, the --include-push-tasks argument needed to be removed
from the cron decision commandline. To set the needed "existing_tasks" parameter,
introduce CRON_OPTIONS in comm_taskgraph.decision. For supported
target_tasks_methods, a parameters key and a callable accepting (parameters,
graph_config) arguments are iterated over, the callable executed, and
parameters[key] set with the result.

Depends on D157846

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c7986566b741
Follow-up [1/2]: Move get_decision_parameters into decision.py r=aleca
https://hg.mozilla.org/comm-central/rev/418ec154506f
Follow-up [2/2]: Fix Daily build cron decision task. r=aleca