Closed Bug 1232005 Opened 6 years ago Closed 5 years ago

Let the gecko decision task generate a file with all the possible jobs that could be scheduled

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(firefox49 fixed)

RESOLVED FIXED
mozilla49
Tracking Status
firefox49 --- fixed

People

(Reporter: armenzg, Assigned: martianwars)

References

Details

Attachments

(2 files, 3 obsolete files)

I proposed [1] having a completed graph generated by the decision task.
This is useful to schedule tasks which are not scheduled on its first execution.
This is necessary to let mozci schedule tasks on behalf of the users.

We will initially start by simply generating a all_tasks.json (better than the "menu" name simply it is clear what it contains).

xenny will be helping us here.

[1] https://groups.google.com/forum/#!topic/mozilla.tools.taskcluster/H0M3nOVJD6c
1. Is the list of all possible jobs that *could* be schedules related to this:
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py

2. Where is the code related to the gecko decision task present?

To generate all_tasks.json, my understanding is that we need a mapping between every build and the set of jobs related to that build. So, maybe using allthethings [1] can be used to find this mapping?
Or, is this sort of a mapping *already* available somewhere? 

[1]: http://mozilla-ci-tools.readthedocs.org/en/latest/allthethings.html
Hi xenny,
In this project, we're to ignore Buildbot completely.
On every developer's push, there are jobs added by Buildbot and a different set of jobs by TaskCluster:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=tc&group_state=expanded

The set of jobs scheduled by TC are defined by a gecko decision task (every push has one):
https://tools.taskcluster.net/task-inspector/#GRY0NAXsQS6WWzZrSrJSBA/0

The jobs you see in Treeherder that are scheduled by TaskCluster as a group of tasks:
https://tools.taskcluster.net/task-graph-inspector/#UGde1ebuRx2hSDANZ0hLAw/ (You can also see the gecko decision task there).

The relationship between builds and tests can be seen in the graph.json file that the decision task generates:
https://public-artifacts.taskcluster.net/GRY0NAXsQS6WWzZrSrJSBA/0/public/graph.json

You will see that some tasks, will have this:
"requires": ["JvuzdYIFSmqS32-USdQ9tA"],

That establishes that a task requires another one to complete first. In our case, test tasks depend on build tasks.

In this first bug, all I need you to focus on is to make the decision task create both a graph.json (as it currently does) plus another artifact called all_tasks.json.
all_tasks.json is always a super set of graph.json.

At the moment, this will *only* be valuable for the try repository where all tasks are *not* scheduled.

After this part completes, we will have to teach mozci how to get a hold of the file all_tasks.json and help other tools schedule tasks based on it.

Please feel free to ask the same questions if it wasn't clear :)
I think this should just be possible by running the mach target a second time in the decision task to create all_tasks.json and use "try: -b do -p all -u all" as the "--message" passed in, instead of the one from the try commit message.

Like we do here for just graph.json: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#77
@garndt: So, I guess I must just modify try.yml as you say, which will create an artifact, along with graph.json in the same directory.
Can you please tell me how the values {{comment}}, {{{url}}}, {{project}}, etc retrieved? I can see owner on this line - [1]. I want to know how to pass the --message as "try: -b do -p all -u all".

[1]: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#23
You should just be able to call the mach command again like this:
            ./mach taskcluster-graph
            --pushlog-id='{{pushlog_id}}'
            --project='{{project}}'
            --message='try: -b do -p all -u all'
            --owner='{{owner}}'
            --revision-hash='{{revision_hash}}'
            --extend-graph > /home/worker/artifacts/all_tasks.json


Those values come from the mozilla-taskcluster when it creates the decision task:
https://github.com/taskcluster/mozilla-taskcluster/blob/02ad893eb27f70a0d968cbb5adb796c0e5c1a949/src/jobs/taskcluster_graph.js#L99-L110
Blocks: 1246167
xenny: hi xenny, do you think you will have time to work on this during March?
No longer blocks: 1246167
Blocks: 1254325
Assignee: vstulsyan → kalpeshk2011
Attached patch all_tasks.patchSplinter Review
I haven't tried pushing this to try server due to Bug 1269446. I hope to try this soon :) Do I need to make any other changes here?
Attachment #8748194 - Flags: review?(armenzg)
Comment on attachment 8748194 [details] [diff] [review]
all_tasks.patch

It looks good in theory but I would like to see it running on try.

garndt: what would be needed to put this on the index?
What could martianwars do to put it on a garbage index while he develops it?
Attachment #8748194 - Flags: review?(armenzg) → review?(garndt)
It appears that Rock has already added an indexed route here:
https://hg.mozilla.org/mozilla-central/rev/52f996dacd8c


If that isn't suitable for what you're looking for, perhaps we could also have an indexed route that includes the revision rather than "latest"
This seems ok, but as Armen pointed out, we should see it run on try.
https://tools.taskcluster.net/task-inspector/#PzmhiGZ8TMiLxeNf0IZtbg/0
Is this okay?
Flags: needinfo?(garndt)
Flags: needinfo?(armenzg)
This looks good.  You didn't include try flags in your commit message so the decision task produced an empty graph like it should, and all_tasks that has 418 tasks in it.  If you want this to be with pushes to branches other than try, you should make the same changes to branch.yml in that same directory.  Nice work.
Flags: needinfo?(garndt)
Disregard the part about branch.yml, that already schedules everything for a branch so no need to generate another file. Try is the one that will filter out jobs, making all_tasks.json useful.
Attachment #8748194 - Flags: review?(garndt) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/a1c5514e09828f86917715f686e5dc9681446903
Bug 1232005 - TaskCluster's Gecko decision task generate a file with all the possible jobs that could be scheduled. r=garndt
Removing NI.
Flags: needinfo?(armenzg)
https://hg.mozilla.org/mozilla-central/rev/a1c5514e0982
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
We should have the artifact being produced for all repositories.
Right now, I don't see anywhere for m-c [1][2], however, I see it working for try [3]

My apologies I should have noticed this earlier.

Currently, we schedule *all* TC jobs on every other repository except try.
However, in the future, we will need this as A) coalescing will be in place and B) we will also have jobs scheduled based on the changes in a push (e.g. we won't trigger mobile jobs if the changes only affect desktop).

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=gecko&selectedJob=3866852&revision=c3f5e6079284
[2] https://tools.taskcluster.net/task-inspector/#C4UJf0PhSleAkNDckZENvg/0
[3] https://tools.taskcluster.net/task-inspector/#CRj-D3JCQBaTTf2FTH9Eow/0
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attached patch alltasks2.patch (obsolete) — Splinter Review
Attachment #8751868 - Flags: review?(armenzg)
Comment on attachment 8751868 [details] [diff] [review]
alltasks2.patch

Review of attachment 8751868 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/tasks/decision/branch.yml
@@ +81,5 @@
>              --owner='{{owner}}'
>              --level='{{level}}'
>              --revision-hash='{{revision_hash}}'
>              --extend-graph > /home/worker/artifacts/graph.json
> +            ./mach taskcluster-graph

There is no reason to run this command again.  the graph produced on a branch is already scheduling everything that can be scheduled for that branch.

If you want a stable file name on all branches to look at (like all_tasks.json) then I would just copy /home/worker/artifacts/graph.json to /home/worker/artifacts/all_tasks.json
Hi garndt, at the moment it will be the same, however, once we have coalescing it won't be the same.
OK to proceed taking this into account?

On another note, is there a better way to compare changes like this? [1] I thought dustin might have something to help with this.

[1]
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph --pushlog-id=30251 --project=mozilla-central '--message= ' --owner=cbook@mozilla.com --level=3 --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
cannot query vcs info because vcs info not provided
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u all' --owner=cbook@mozilla.com --level=3 --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
cannot query vcs info because vcs info not provided
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2
Attached patch alltasks2.patch (obsolete) — Splinter Review
Is this better now?
Attachment #8751868 - Attachment is obsolete: true
Attachment #8751868 - Flags: review?(armenzg)
Attachment #8752173 - Flags: review?(garndt)
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #20)
> Hi garndt, at the moment it will be the same, however, once we have
> coalescing it won't be the same.
> OK to proceed taking this into account?
> 
> On another note, is there a better way to compare changes like this? [1] I
> thought dustin might have something to help with this.
> 
> [1]
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message= '
> --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u
> all' --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2

with the existing graph command, appending "--dry-run" but even then it's not the easiest.  I think with the new graph commands it will be *much* simpler to do these things and I believe he has thought about having a mach command specifically for showing a diff of before and after of a graph.
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #20)
> Hi garndt, at the moment it will be the same, however, once we have
> coalescing it won't be the same.
> OK to proceed taking this into account?
> 
> On another note, is there a better way to compare changes like this? [1] I
> thought dustin might have something to help with this.
> 
> [1]
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message= '
> --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u
> all' --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2

re: coalescing
I believe (and I could be wrong) that coalescing in taskcluster will still result in the decision graph scheduling everything, but a service will have a worker coalesce tasks after they have been scheduled.
(In reply to Kalpesh Krishna [:martianwars] from comment #21)
> Created attachment 8752173 [details] [diff] [review]
> alltasks2.patch
> 
> Is this better now?

I think it's ok, but I'll leave it up to armen to see if this fits into what you two are attempting to accomplish.
Hey Jake, how does coalescing work?
Do all tasks get scheduled with coalescer but a service will have the worker coalesce the tasks?
In other words, would graph.json be mainly the same (when it comes to noumber of tasks) with and without coalescer?

Thanks!

(In reply to Greg Arndt [:garndt] from comment #23)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #20)
> > Hi garndt, at the moment it will be the same, however, once we have
> > coalescing it won't be the same.
> > 
> re: coalescing
> I believe (and I could be wrong) that coalescing in taskcluster will still
> result in the decision graph scheduling everything, but a service will have
> a worker coalesce tasks after they have been scheduled.
Flags: needinfo?(jwatkins)
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #25)
> Hey Jake, how does coalescing work?
> Do all tasks get scheduled with coalescer but a service will have the worker
> coalesce the tasks?
> In other words, would graph.json be mainly the same (when it comes to
> noumber of tasks) with and without coalescer?
> 
> Thanks!

:garndt is correct.  Right now (although not enabled in prod just yet), the coalesce service simply builds lists of tasks (by listening to pulse) that can be coalesced and serves that list to a requesting worker.  The worker does the coalescing by claiming the task[0] and resolving tasks[0+n] as completed with exception.  Obviously, task[0] is the more recent of all the tasks.  So the graph will not change.
Flags: needinfo?(jwatkins)
Comment on attachment 8752173 [details] [diff] [review]
alltasks2.patch

Review of attachment 8752173 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks dividehex!

In that case, we can land this.

@martianwars: does this work for you? do you want me to land it for you?
Attachment #8752173 - Flags: review?(garndt) → review+
armenzg, works for me, let's land this :D
https://hg.mozilla.org/integration/mozilla-inbound/rev/dd29998a7d44c0caa79881afe494024890b91676
Bug 1232005 - Generating all_tasks.json in gecko decision task for all branches. DONTBUILD. CLOSED TREE. r=armenzg
It appears that there might be an '&&' missed after generating the graph and copying it.  This would result in an error similar to here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e0c637d330f&selectedJob=20906139
Attached patch alltasks2.patch (obsolete) — Splinter Review
Sorry for the really silly mistake
Attachment #8752173 - Attachment is obsolete: true
Attachment #8753051 - Flags: review?(garndt)
Attached patch alltasks2.patchSplinter Review
sorry again :(
Attachment #8753051 - Attachment is obsolete: true
Attachment #8753051 - Flags: review?(garndt)
Attachment #8753055 - Flags: review?(garndt)
Comment on attachment 8753055 [details] [diff] [review]
alltasks2.patch

Review of attachment 8753055 [details] [diff] [review]:
-----------------------------------------------------------------

r+ but there's nothing to do with this patch.  I believe Armen pushed a change already.
Attachment #8753055 - Flags: review?(garndt) → review+
You need to log in before you can comment on or make changes to this bug.