1232005 - Let the gecko decision task generate a file with all the possible jobs that could be scheduled

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Armen [:armenzg]]

I proposed [1] having a completed graph generated by the decision task.
This is useful to schedule tasks which are not scheduled on its first execution.
This is necessary to let mozci schedule tasks on behalf of the users.

We will initially start by simply generating a all_tasks.json (better than the "menu" name simply it is clear what it contains).

xenny will be helping us here.

[1] https://groups.google.com/forum/#!topic/mozilla.tools.taskcluster/H0M3nOVJD6c

Comment 1

[Vaibhav Tulsyan [:xenny]]

1. Is the list of all possible jobs that *could* be schedules related to this:
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py

2. Where is the code related to the gecko decision task present?

To generate all_tasks.json, my understanding is that we need a mapping between every build and the set of jobs related to that build. So, maybe using allthethings [1] can be used to find this mapping?
Or, is this sort of a mapping *already* available somewhere? 

[1]: http://mozilla-ci-tools.readthedocs.org/en/latest/allthethings.html

Comment 2

[Armen [:armenzg]]

Hi xenny,
In this project, we're to ignore Buildbot completely.
On every developer's push, there are jobs added by Buildbot and a different set of jobs by TaskCluster:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=tc&group_state=expanded

The set of jobs scheduled by TC are defined by a gecko decision task (every push has one):
https://tools.taskcluster.net/task-inspector/#GRY0NAXsQS6WWzZrSrJSBA/0

The jobs you see in Treeherder that are scheduled by TaskCluster as a group of tasks:
https://tools.taskcluster.net/task-graph-inspector/#UGde1ebuRx2hSDANZ0hLAw/ (You can also see the gecko decision task there).

The relationship between builds and tests can be seen in the graph.json file that the decision task generates:
https://public-artifacts.taskcluster.net/GRY0NAXsQS6WWzZrSrJSBA/0/public/graph.json

You will see that some tasks, will have this:
"requires": ["JvuzdYIFSmqS32-USdQ9tA"],

That establishes that a task requires another one to complete first. In our case, test tasks depend on build tasks.

In this first bug, all I need you to focus on is to make the decision task create both a graph.json (as it currently does) plus another artifact called all_tasks.json.
all_tasks.json is always a super set of graph.json.

At the moment, this will *only* be valuable for the try repository where all tasks are *not* scheduled.

After this part completes, we will have to teach mozci how to get a hold of the file all_tasks.json and help other tools schedule tasks based on it.

Please feel free to ask the same questions if it wasn't clear :)

Comment 3

[Greg Arndt [:garndt]]

I think this should just be possible by running the mach target a second time in the decision task to create all_tasks.json and use "try: -b do -p all -u all" as the "--message" passed in, instead of the one from the try commit message.

Like we do here for just graph.json: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#77

Comment 4

[Vaibhav Tulsyan [:xenny]]

@garndt: So, I guess I must just modify try.yml as you say, which will create an artifact, along with graph.json in the same directory.
Can you please tell me how the values {{comment}}, {{{url}}}, {{project}}, etc retrieved? I can see owner on this line - [1]. I want to know how to pass the --message as "try: -b do -p all -u all".

[1]: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#23

Comment 5

[Greg Arndt [:garndt]]

You should just be able to call the mach command again like this:
            ./mach taskcluster-graph
            --pushlog-id='{{pushlog_id}}'
            --project='{{project}}'
            --message='try: -b do -p all -u all'
            --owner='{{owner}}'
            --revision-hash='{{revision_hash}}'
            --extend-graph > /home/worker/artifacts/all_tasks.json


Those values come from the mozilla-taskcluster when it creates the decision task:
https://github.com/taskcluster/mozilla-taskcluster/blob/02ad893eb27f70a0d968cbb5adb796c0e5c1a949/src/jobs/taskcluster_graph.js#L99-L110

Comment 6

[Armen [:armenzg]]

xenny: hi xenny, do you think you will have time to work on this during March?

Comment 7

[Kalpesh Krishna [:martianwars]]

Attachment: all_tasks.patch

I haven't tried pushing this to try server due to Bug 1269446. I hope to try this soon :) Do I need to make any other changes here?

Comment 8

[Armen [:armenzg]]

Comment on attachment 8748194 [details] [diff] [review]
all_tasks.patch

It looks good in theory but I would like to see it running on try.

garndt: what would be needed to put this on the index?
What could martianwars do to put it on a garbage index while he develops it?

Comment 9

[Greg Arndt [:garndt]]

It appears that Rock has already added an indexed route here:
https://hg.mozilla.org/mozilla-central/rev/52f996dacd8c


If that isn't suitable for what you're looking for, perhaps we could also have an indexed route that includes the revision rather than "latest"

Comment 10

[Greg Arndt [:garndt]]

This seems ok, but as Armen pointed out, we should see it run on try.

Comment 11

[Kalpesh Krishna [:martianwars]]

https://tools.taskcluster.net/task-inspector/#PzmhiGZ8TMiLxeNf0IZtbg/0
Is this okay?

Comment 12

[Greg Arndt [:garndt]]

This looks good.  You didn't include try flags in your commit message so the decision task produced an empty graph like it should, and all_tasks that has 418 tasks in it.  If you want this to be with pushes to branches other than try, you should make the same changes to branch.yml in that same directory.  Nice work.

Comment 13

[Greg Arndt [:garndt]]

Disregard the part about branch.yml, that already schedules everything for a branch so no need to generate another file. Try is the one that will filter out jobs, making all_tasks.json useful.

Comment 14

[Armen [:armenzg]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a1c5514e09828f86917715f686e5dc9681446903
Bug 1232005 - TaskCluster's Gecko decision task generate a file with all the possible jobs that could be scheduled. r=garndt

Comment 15

[Armen [:armenzg]]

Removing NI.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a1c5514e0982

Comment 17

[Armen [:armenzg]]

We should have the artifact being produced for all repositories.
Right now, I don't see anywhere for m-c [1][2], however, I see it working for try [3]

My apologies I should have noticed this earlier.

Currently, we schedule *all* TC jobs on every other repository except try.
However, in the future, we will need this as A) coalescing will be in place and B) we will also have jobs scheduled based on the changes in a push (e.g. we won't trigger mobile jobs if the changes only affect desktop).

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=gecko&selectedJob=3866852&revision=c3f5e6079284
[2] https://tools.taskcluster.net/task-inspector/#C4UJf0PhSleAkNDckZENvg/0
[3] https://tools.taskcluster.net/task-inspector/#CRj-D3JCQBaTTf2FTH9Eow/0

Comment 18

[Kalpesh Krishna [:martianwars]]

Attachment: alltasks2.patch

Comment 19

[Greg Arndt [:garndt]]

Comment on attachment 8751868 [details] [diff] [review]
alltasks2.patch

Review of attachment 8751868 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/tasks/decision/branch.yml
@@ +81,5 @@
>              --owner='{{owner}}'
>              --level='{{level}}'
>              --revision-hash='{{revision_hash}}'
>              --extend-graph > /home/worker/artifacts/graph.json
> +            ./mach taskcluster-graph

There is no reason to run this command again.  the graph produced on a branch is already scheduling everything that can be scheduled for that branch.

If you want a stable file name on all branches to look at (like all_tasks.json) then I would just copy /home/worker/artifacts/graph.json to /home/worker/artifacts/all_tasks.json

Comment 20

[Armen [:armenzg]]

Hi garndt, at the moment it will be the same, however, once we have coalescing it won't be the same.
OK to proceed taking this into account?

On another note, is there a better way to compare changes like this? [1] I thought dustin might have something to help with this.

[1]
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph --pushlog-id=30251 --project=mozilla-central '--message= ' --owner=[e-mail] --level=3 --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
cannot query vcs info because vcs info not provided
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u all' --owner=[e-mail] --level=3 --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
cannot query vcs info because vcs info not provided
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2

Comment 21

[Kalpesh Krishna [:martianwars]]

Attachment: alltasks2.patch

Is this better now?

Comment 22

[Greg Arndt [:garndt]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #20)
> Hi garndt, at the moment it will be the same, however, once we have
> coalescing it won't be the same.
> OK to proceed taking this into account?
> 
> On another note, is there a better way to compare changes like this? [1] I
> thought dustin might have something to help with this.
> 
> [1]
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message= '
> --owner=[e-mail] --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u
> all' --owner=[e-mail] --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2

with the existing graph command, appending "--dry-run" but even then it's not the easiest.  I think with the new graph commands it will be *much* simpler to do these things and I believe he has thought about having a mach command specifically for showing a diff of before and after of a graph.

Comment 23

[Greg Arndt [:garndt]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #20)
> Hi garndt, at the moment it will be the same, however, once we have
> coalescing it won't be the same.
> OK to proceed taking this into account?
> 
> On another note, is there a better way to compare changes like this? [1] I
> thought dustin might have something to help with this.
> 
> [1]
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message= '
> --owner=[e-mail] --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u
> all' --owner=[e-mail] --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2

re: coalescing
I believe (and I could be wrong) that coalescing in taskcluster will still result in the decision graph scheduling everything, but a service will have a worker coalesce tasks after they have been scheduled.

Comment 24

[Greg Arndt [:garndt]]

(In reply to Kalpesh Krishna [:martianwars] from comment #21)
> Created attachment 8752173 [details] [diff] [review]
> alltasks2.patch
> 
> Is this better now?

I think it's ok, but I'll leave it up to armen to see if this fits into what you two are attempting to accomplish.

Comment 25

[Armen [:armenzg]]

Hey Jake, how does coalescing work?
Do all tasks get scheduled with coalescer but a service will have the worker coalesce the tasks?
In other words, would graph.json be mainly the same (when it comes to noumber of tasks) with and without coalescer?

Thanks!

(In reply to Greg Arndt [:garndt] from comment #23)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #20)
> > Hi garndt, at the moment it will be the same, however, once we have
> > coalescing it won't be the same.
> > 
> re: coalescing
> I believe (and I could be wrong) that coalescing in taskcluster will still
> result in the decision graph scheduling everything, but a service will have
> a worker coalesce tasks after they have been scheduled.

Comment 26

[Jake Watkins [:dividehex]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #25)
> Hey Jake, how does coalescing work?
> Do all tasks get scheduled with coalescer but a service will have the worker
> coalesce the tasks?
> In other words, would graph.json be mainly the same (when it comes to
> noumber of tasks) with and without coalescer?
> 
> Thanks!

:garndt is correct.  Right now (although not enabled in prod just yet), the coalesce service simply builds lists of tasks (by listening to pulse) that can be coalesced and serves that list to a requesting worker.  The worker does the coalescing by claiming the task[0] and resolving tasks[0+n] as completed with exception.  Obviously, task[0] is the more recent of all the tasks.  So the graph will not change.

Comment 27

[Armen [:armenzg]]

Comment on attachment 8752173 [details] [diff] [review]
alltasks2.patch

Review of attachment 8752173 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks dividehex!

In that case, we can land this.

@martianwars: does this work for you? do you want me to land it for you?

Comment 28

[Kalpesh Krishna [:martianwars]]

armenzg, works for me, let's land this :D

Comment 29

[Armen [:armenzg]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/dd29998a7d44c0caa79881afe494024890b91676
Bug 1232005 - Generating all_tasks.json in gecko decision task for all branches. DONTBUILD. CLOSED TREE. r=armenzg

Comment 30

[Greg Arndt [:garndt]]

It appears that there might be an '&&' missed after generating the graph and copying it.  This would result in an error similar to here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e0c637d330f&selectedJob=20906139

Comment 31

[Kalpesh Krishna [:martianwars]]

Attachment: alltasks2.patch

Sorry for the really silly mistake

Comment 32

[Kalpesh Krishna [:martianwars]]

Attachment: alltasks2.patch

sorry again :(

Comment 33

[Armen [:armenzg]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4dab7153d8470a6f0758b8c9e8cd46515b9f3d8d
Bug 1232005 - Fix typo in dd29998a7d44. CLOSED TREE. r=garndt

Comment 34

[Greg Arndt [:garndt]]

Comment on attachment 8753055 [details] [diff] [review]
alltasks2.patch

Review of attachment 8753055 [details] [diff] [review]:
-----------------------------------------------------------------

r+ but there's nothing to do with this patch.  I believe Armen pushed a change already.

Comment 35

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/dd29998a7d44
https://hg.mozilla.org/mozilla-central/rev/4dab7153d847