1174313 - [gecko] Create mach target for generating list of post jobs for a given branch

Status: RESOLVED WORKSFORME

(Taskcluster :: General, defect)

Description

[Greg Arndt [:garndt]]

This might live as a poc or expand into something more but...

There is some value in generating a list of tasks similar to all_builders.txt [1] that could be used when selecting what tasks to submit.  Initially this could be used for mozci to build the try syntax to pass to `mach taskcluster-graph`.

[1] http://people.mozilla.org/~armenzg/permanent/all_builders.txt

Comment 1

[Greg Arndt [:garndt]]

Attachment: MozReview Request: Bug 1174313 - Mach target for building job list for all/specific projects

Bug 1174313 - Mach target for building job list for all/specific projects

Comment 2

[Greg Arndt [:garndt]]

Comment on attachment 8622647 [details]
MozReview Request: Bug 1174313 - Mach target for building job list for all/specific projects

Here is a quick hacky-please-dont-judge-me mach target for generating a list of jobs based on our in tree job flags.  Either you can pass in '--project' and limit the output, or don't pass it in and get everything.  By default our mach taskcluster-graph target will take any project, and if we don't have a branch specific config, it will default to a base set of jobs.

I think this format matches the example you showed me.

Here is the output of not passing in --project:
https://gist.github.com/gregarndt/9dadb0ad43e10508aabf

Comment 3

[Armen [:armenzg]]

https://reviewboard.mozilla.org/r/11423/#review9787

::: testing/taskcluster/mach_commands.py:470
(Diff revision 1)
> +    def create_job_list(self, **params):

Add project=None to the signature.

Could you also please add a doc string under the function?

Comment 4

[Armen [:armenzg]]

The output looks good.

However, I'm still trying to figure out how to use to make it more useful for the scheduling work we're looking into.

For instance, this job will not have any jobs triggered in taskcluster:
https://treeherder.allizom.org/#/jobs?repo=try&revision=1b891687a19f

If I want to schedule "linux64_gecko try opt test gaia-build-unit", how could I map it to something that will create the correct task graph?

Comment 5

[Armen [:armenzg]]

Comment on attachment 8622647 [details]
MozReview Request: Bug 1174313 - Mach target for building job list for all/specific projects

https://reviewboard.mozilla.org/r/11423/#review9897

Ship It!

Comment 6

[Armen [:armenzg]]

Comment on attachment 8622647 [details]
MozReview Request: Bug 1174313 - Mach target for building job list for all/specific projects

I realized that the **params is the way mach passes parameters into a function so ignore what I said.

Comment 7

[Greg Arndt [:garndt]]

> If I want to schedule "linux64_gecko try opt test gaia-build-unit", how
> could I map it to something that will create the correct task graph?

so that output is <build flag> <project> <build type> <build or test> <test flag>

this would translate in a try command:
try -b o -p linux64_gecko -u gaia-build-unit

I tried it out, and here are my steps.  This is somewhat painful so I hope that we can work on making it better:
1. Pushed to try with flags that didn't include building b2g-desktop and running gaia-build-unit...
2. Remember that oh wait, I want to build b2gdesktop, so I took the hg commit hash (3d4efab3bc24) that was pushed to try and looked it up here: https://treeherder.mozilla.org/api/project/try/revision-lookup/?revision=3d4efab3bc24
3. Used the revision_hash returned by that to construct the taskcluster-graph command I need:
mach taskcluster-graph --project try --message "try: -b o -p linux64_gecko -u gaia-build-unit" --base-repository http://hg.mozilla.org/mozilla-central  --head-repository http://hg.mozilla.org/try --head-rev 3d4efab3bc24 --owner [e-mail] --revision-hash 4504e49c9b5d2d5f4ec2d2945e35652a03cc66cc | taskcluster run-graph --verbose

Here you can see my try push that didn't trigger the b2g build, and then I added the jobs by using `mach taskcluster-graph`
https://treeherder.allizom.org/#/jobs?repo=try&revision=3d4efab3bc24

Comment 8

[Armen [:armenzg]]

garndt, this is exactly what I had in mind!

I will make it easier and propose an API to help building tools for this purpose.
We should review this together to see what makes sense for such API.

Comment 9

[Armen [:armenzg]]

Why does --base-repository point to mozilla-central instead of try?
What was the --revision-hash again?

This is the error that I get.

MacAir mozilla-inbound hg:[default!] $ ./mach taskcluster-graph --project try --message "try: -b o -p linux64_gecko -u gaia-build-unit" --base-repository http://hg.mozilla.org/mozilla-central  --head-repository http://hg.mozilla.org/try --head-rev 1b891687a19f --owner [e-mail] --revision-hash  17e1e266875bf3eda8d80008e3ccb0a2d7796d49 | taskcluster run-graph --verbose
Taskgraph creation error Error: Authorization Failed
{
  "message": "Authorization Failed",
  "error": {
    "info": "None of the scope-sets was satisfied",
    "scopesets": [
      [
        "queue:route:tc-treeherder-stage.try.17e1e266875bf3eda8d80008e3ccb0a2d7796d49",
        "docker-worker:capability:device:loopbackVideo",
        "queue:create-task:aws-provisioner-v1/test-c4-2xlarge",
        "queue:create-task:aws-provisioner-v1/build-c4-2xlarge",
        "docker-worker:image:taskcluster/tester:0.3.5",
        "queue:route:index.buildbot.revisions.1b891687a19f.try.linux64_gecko",
        "queue:define-task:aws-provisioner-v1/test-c4-2xlarge",
        "queue:define-task:aws-provisioner-v1/build-c4-2xlarge",
        "queue:route:index.buildbot.branches.try.linux64_gecko",
        "queue:route:index.gecko.v1.try.revision.linux.1b891687a19f.b2g-desktop.opt",
        "docker-worker:cache:workspace-b2g-desktop-objects-opt",
        "docker-worker:image:taskcluster/builder:0.5.5",
        "docker-worker:cache:tc-vcs",
        "queue:route:index.gecko.v1.try.latest.linux.b2g-desktop.opt",
        "queue:define-task:aws-provisioner-v1/b2gtest",
        "queue:route:tc-treeherder.try.17e1e266875bf3eda8d80008e3ccb0a2d7796d49",
        "docker-worker:capability:device:loopbackAudio",
        "docker-worker:cache:linux-cache",
        "queue:define-task:aws-provisioner-v1/b2g-desktop-opt"
      ]
    ]
  }
}

Comment 10

[Greg Arndt [:garndt]]

regarding base-repository, this was taken from our decision task logic [1].  My understanding is that try pushes are pushes applied on top of m-c, but perhaps I'm mistaken about that.

re: authorization failure, I think this is because of the scopes for device capabilities.  Since they are just loopback devices, and part of every task, I added the scopes for temporary credentials.  These changes can take up to an hour to propagate and you will need to log out and back in again I think. 

Let me know if there is anything else I can help with. 

[1] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#48

Comment 11

[Armen [:armenzg]]

I'm thinking of closing this bug for now and re-open it once we have a clear plan (to be discussed on Whistler).
Thanks a lot for your help and for that patch! It's pretty useful and I think it will very useful once I know how to use the generated data.

What do you think?

Comment 12

[Armen [:armenzg]]

and:
> What was the --revision-hash again?

Is that an internal value to treeherder?

Comment 13

[Greg Arndt [:garndt]]

Yup, I agree that we could close this now.  --revision-hash is a hash that treeherder generates for a set of jobs (at least that's what I remember).

mozilla-taskcluster has some code for calculating it, which I think is a copy from treeherder:
https://github.com/taskcluster/mozilla-taskcluster/blob/master/src/treeherder/resultset.js#L15-40

Comment 14

[Armen [:armenzg]]

We decided not to land this.
Nevertheless, it was very informative and helped us understand the issue better!

Thanks Greg!

Comment 15

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...

Comment 16

[Armen [:armenzg]]

This is actually useful for me to generate the data necessary to determine what new jobs we're adding after some code changes.

Comment 17

[Armen [:armenzg]]

Not working on this at the moment.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

I think this is 100% bitrotted, and probably irrelevant, after the in-tree taskgraph stuff.