1410513 - Teach taskgraph to get try details for try-comm-central pushes.

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect, P2)

Description

[Tom Prince [:tomprince]]

Currently it looks at the revisions of mozilla-central to get try-syntax from.

Comment 1

[Tom Prince [:tomprince]]

Attachment: Bug 1410513: Use try_tasks as target method on try-comm-central;

Review commit: https://reviewboard.mozilla.org/r/195980/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/195980/

Comment 2

[Tom Prince [:tomprince]]

Attachment: Bug 1410513: Treeherder group names from config file;

Review commit: https://reviewboard.mozilla.org/r/195982/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/195982/

Comment 3

[Tom Prince [:tomprince]]

Attachment: Bug 1410513: Move ridealong build configuration to ci/config.yml;

Review commit: https://reviewboard.mozilla.org/r/195984/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/195984/

Comment 4

[Tom Prince [:tomprince]]

Attachment: Bug 1410513: Add treeherder groupnames to taskgraph configuration;

Review commit: https://reviewboard.mozilla.org/r/196004/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/196004/

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8924758 [details]
Bug 1410513: Use try_tasks as target method on try-comm-central;

https://reviewboard.mozilla.org/r/195980/#review201448

::: taskcluster/taskgraph/decision.py:208
(Diff revision 1)
>  
>      # if try_task_config.json is present, load it
>      task_config_file = os.path.join(os.getcwd(), 'try_task_config.json')
>  
>      # load try settings
> -    if project == 'try':
> +    if 'try' in project:

maybe `parameters['target_tasks_method'] == 'try_tasks'` instead?  What if we make a project branch named `ostrya`?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8924759 [details]
Bug 1410513: Treeherder group names from config file;

https://reviewboard.mozilla.org/r/195982/#review201460

I like this -- it is a good solution for data in that awkward place between "yaml job description" and code.

::: taskcluster/taskgraph/config.py:7
(Diff revision 1)
> +from voluptuous import Required
> +
> +graph_config_schema = Schema({
> +    Required('treeherder'): {
> +        # Mapping of treeherder group symbols to descriptive names
> +        Required('group-names'): {basestring: basestring}

++ for adding a schema :)

::: taskcluster/taskgraph/generator.py:213
(Diff revision 1)
> +            config = yaml.load(f)
> +
> +        return validate_graph_config(config)
>  
>      def _run(self):
> +        logger.info("Loading global configuration.")

"graph configuration"?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8924760 [details]
Bug 1410513: Move ridealong build configuration to ci/config.yml;

https://reviewboard.mozilla.org/r/195984/#review201464

::: taskcluster/taskgraph/config.py:2
(Diff revision 1)
>  from .util.schema import validate_schema, Schema
> -from voluptuous import Required
> +from voluptuous import Required, Optional

unused?

::: taskcluster/taskgraph/try_option_syntax.py
(Diff revision 1)
> -        'sm-asan',
> -        'sm-mozjs-sys',
> -        'sm-msan',
> -        'sm-fuzzing',
> -        'sm-rust-bindings',
> -    ],

note that this will likely conflict with bug 1413721

::: taskcluster/taskgraph/try_option_syntax.py:327
(Diff revision 1)
>  
>      def parse_platforms(self, platform_arg, full_task_graph):
>          if platform_arg == 'all':
>              return None
>  
> +        RIDEALONG_BUILDS = self.graph_config['ridealong-builds']

I think I'd rather see `self.graph_config['try']['ridealong-builds']` here, otherwise "graph config" is a subtree of itself.

Comment 8

[Tom Prince [:tomprince]]

Comment on attachment 8924758 [details]
Bug 1410513: Use try_tasks as target method on try-comm-central;

https://reviewboard.mozilla.org/r/195980/#review201448

> maybe `parameters['target_tasks_method'] == 'try_tasks'` instead?  What if we make a project branch named `ostrya`?

That was what I originally had, but then I noticed that there are a few places (such as [here](https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/build.py#49)) that look at these parameters even if `target_task_method` isn't `try_tasks`.

It is the same logic which I used [here](https://reviewboard.mozilla.org/r/191262/diff/1#index_header), for which I filed [Bug 1410255](https://bugzilla.mozilla.org/show_bug.cgi?id=1410255). Given the above and that but, I think it makes sense to keep the condition the same, so that it'll be easier to find the conditions when 1410255 is dealt with.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

fair enough!

Comment 10

[Tom Prince [:tomprince]]

Comment on attachment 8924758 [details]
Bug 1410513: Use try_tasks as target method on try-comm-central;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195980/diff/1-2/

Comment 11

[Tom Prince [:tomprince]]

Comment on attachment 8924759 [details]
Bug 1410513: Treeherder group names from config file;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195982/diff/1-2/

Comment 12

[Tom Prince [:tomprince]]

Comment on attachment 8924760 [details]
Bug 1410513: Move ridealong build configuration to ci/config.yml;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195984/diff/1-2/

Comment 13

[Tom Prince [:tomprince]]

Comment on attachment 8924758 [details]
Bug 1410513: Use try_tasks as target method on try-comm-central;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195980/diff/2-3/

Comment 14

[Tom Prince [:tomprince]]

Comment on attachment 8924759 [details]
Bug 1410513: Treeherder group names from config file;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195982/diff/2-3/

Comment 15

[Tom Prince [:tomprince]]

Comment on attachment 8924760 [details]
Bug 1410513: Move ridealong build configuration to ci/config.yml;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195984/diff/2-3/

Comment 16

[Tom Prince [:tomprince]]

Comment on attachment 8924759 [details]
Bug 1410513: Treeherder group names from config file;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195982/diff/3-4/

Comment 17

[Tom Prince [:tomprince]]

Comment on attachment 8924760 [details]
Bug 1410513: Move ridealong build configuration to ci/config.yml;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195984/diff/3-4/

Comment 18

[Mozilla Autoland]

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 12f72395c780 -d 516690579442: rebasing 432063:12f72395c780 "Bug 1410513: Use try_tasks as target method on try-comm-central; r=dustin"
rebasing 432064:001d81ce67ee "Bug 1410513: Treeherder group names from config file; r=dustin"
merging taskcluster/taskgraph/generator.py
merging taskcluster/taskgraph/test/test_generator.py
merging taskcluster/taskgraph/transforms/task.py
rebasing 432065:4c8fd6a52e35 "Bug 1410513: Move ridealong build configuration to ci/config.yml; r=dustin" (tip)
merging taskcluster/taskgraph/generator.py
merging taskcluster/taskgraph/test/test_generator.py
warning: conflicts while merging taskcluster/taskgraph/test/test_generator.py! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)

Comment 19

[Tom Prince [:tomprince]]

Comment on attachment 8924758 [details]
Bug 1410513: Use try_tasks as target method on try-comm-central;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195980/diff/3-4/

Comment 20

[Tom Prince [:tomprince]]

Comment on attachment 8924759 [details]
Bug 1410513: Treeherder group names from config file;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195982/diff/4-5/

Comment 21

[Tom Prince [:tomprince]]

Comment on attachment 8924760 [details]
Bug 1410513: Move ridealong build configuration to ci/config.yml;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/195984/diff/4-5/

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/713b8f8146a2
Use try_tasks as target method on try-comm-central; r=dustin
https://hg.mozilla.org/integration/autoland/rev/69c60c8fc528
Treeherder group names from config file; r=dustin
https://hg.mozilla.org/integration/autoland/rev/f8903937a76e
Move ridealong build configuration to ci/config.yml; r=dustin

Comment 23

[Tom Prince [:tomprince]]

Comment on attachment 8924769 [details]
Bug 1410513: Add treeherder groupnames to taskgraph configuration;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/196004/diff/1-2/

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/bae7ea206ae9
Add treeherder groupnames to taskgraph configuration; r=me CLOSED TREE DONTBUILD

Comment 25

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/713b8f8146a2
https://hg.mozilla.org/mozilla-central/rev/69c60c8fc528
https://hg.mozilla.org/mozilla-central/rev/f8903937a76e