Let the gecko decision task generate a file with all the possible jobs that could be scheduled

RESOLVED FIXED in Firefox 49

Status

Testing
General
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: armenzg, Assigned: martianwars)

Tracking

unspecified
mozilla49
Points:
---

Firefox Tracking Flags

(firefox49 fixed)

Details

Attachments

(2 attachments, 3 obsolete attachments)

(Reporter)

Description

2 years ago
I proposed [1] having a completed graph generated by the decision task.
This is useful to schedule tasks which are not scheduled on its first execution.
This is necessary to let mozci schedule tasks on behalf of the users.

We will initially start by simply generating a all_tasks.json (better than the "menu" name simply it is clear what it contains).

xenny will be helping us here.

[1] https://groups.google.com/forum/#!topic/mozilla.tools.taskcluster/H0M3nOVJD6c
1. Is the list of all possible jobs that *could* be schedules related to this:
https://github.com/mozilla/treeherder/blob/master/treeherder/etl/buildbot.py

2. Where is the code related to the gecko decision task present?

To generate all_tasks.json, my understanding is that we need a mapping between every build and the set of jobs related to that build. So, maybe using allthethings [1] can be used to find this mapping?
Or, is this sort of a mapping *already* available somewhere? 

[1]: http://mozilla-ci-tools.readthedocs.org/en/latest/allthethings.html
(Reporter)

Comment 2

2 years ago
Hi xenny,
In this project, we're to ignore Buildbot completely.
On every developer's push, there are jobs added by Buildbot and a different set of jobs by TaskCluster:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=tc&group_state=expanded

The set of jobs scheduled by TC are defined by a gecko decision task (every push has one):
https://tools.taskcluster.net/task-inspector/#GRY0NAXsQS6WWzZrSrJSBA/0

The jobs you see in Treeherder that are scheduled by TaskCluster as a group of tasks:
https://tools.taskcluster.net/task-graph-inspector/#UGde1ebuRx2hSDANZ0hLAw/ (You can also see the gecko decision task there).

The relationship between builds and tests can be seen in the graph.json file that the decision task generates:
https://public-artifacts.taskcluster.net/GRY0NAXsQS6WWzZrSrJSBA/0/public/graph.json

You will see that some tasks, will have this:
"requires": ["JvuzdYIFSmqS32-USdQ9tA"],

That establishes that a task requires another one to complete first. In our case, test tasks depend on build tasks.

In this first bug, all I need you to focus on is to make the decision task create both a graph.json (as it currently does) plus another artifact called all_tasks.json.
all_tasks.json is always a super set of graph.json.

At the moment, this will *only* be valuable for the try repository where all tasks are *not* scheduled.

After this part completes, we will have to teach mozci how to get a hold of the file all_tasks.json and help other tools schedule tasks based on it.

Please feel free to ask the same questions if it wasn't clear :)

Comment 3

2 years ago
I think this should just be possible by running the mach target a second time in the decision task to create all_tasks.json and use "try: -b do -p all -u all" as the "--message" passed in, instead of the one from the try commit message.

Like we do here for just graph.json: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#77
@garndt: So, I guess I must just modify try.yml as you say, which will create an artifact, along with graph.json in the same directory.
Can you please tell me how the values {{comment}}, {{{url}}}, {{project}}, etc retrieved? I can see owner on this line - [1]. I want to know how to pass the --message as "try: -b do -p all -u all".

[1]: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/decision/try.yml#23

Comment 5

2 years ago
You should just be able to call the mach command again like this:
            ./mach taskcluster-graph
            --pushlog-id='{{pushlog_id}}'
            --project='{{project}}'
            --message='try: -b do -p all -u all'
            --owner='{{owner}}'
            --revision-hash='{{revision_hash}}'
            --extend-graph > /home/worker/artifacts/all_tasks.json


Those values come from the mozilla-taskcluster when it creates the decision task:
https://github.com/taskcluster/mozilla-taskcluster/blob/02ad893eb27f70a0d968cbb5adb796c0e5c1a949/src/jobs/taskcluster_graph.js#L99-L110
(Reporter)

Updated

2 years ago
Blocks: 1246167
(Reporter)

Comment 6

2 years ago
xenny: hi xenny, do you think you will have time to work on this during March?
(Reporter)

Updated

2 years ago
No longer blocks: 1246167
(Reporter)

Updated

2 years ago
Blocks: 1254325
(Assignee)

Updated

2 years ago
Assignee: vstulsyan → kalpeshk2011
(Assignee)

Comment 7

2 years ago
Created attachment 8748194 [details] [diff] [review]
all_tasks.patch

I haven't tried pushing this to try server due to Bug 1269446. I hope to try this soon :) Do I need to make any other changes here?
Attachment #8748194 - Flags: review?(armenzg)
(Reporter)

Comment 8

2 years ago
Comment on attachment 8748194 [details] [diff] [review]
all_tasks.patch

It looks good in theory but I would like to see it running on try.

garndt: what would be needed to put this on the index?
What could martianwars do to put it on a garbage index while he develops it?
Attachment #8748194 - Flags: review?(armenzg) → review?(garndt)

Comment 9

2 years ago
It appears that Rock has already added an indexed route here:
https://hg.mozilla.org/mozilla-central/rev/52f996dacd8c


If that isn't suitable for what you're looking for, perhaps we could also have an indexed route that includes the revision rather than "latest"
This seems ok, but as Armen pointed out, we should see it run on try.
(Assignee)

Comment 11

2 years ago
https://tools.taskcluster.net/task-inspector/#PzmhiGZ8TMiLxeNf0IZtbg/0
Is this okay?
Flags: needinfo?(garndt)
Flags: needinfo?(armenzg)
This looks good.  You didn't include try flags in your commit message so the decision task produced an empty graph like it should, and all_tasks that has 418 tasks in it.  If you want this to be with pushes to branches other than try, you should make the same changes to branch.yml in that same directory.  Nice work.
Flags: needinfo?(garndt)
Disregard the part about branch.yml, that already schedules everything for a branch so no need to generate another file. Try is the one that will filter out jobs, making all_tasks.json useful.

Updated

2 years ago
Attachment #8748194 - Flags: review?(garndt) → review+
(Reporter)

Comment 14

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/a1c5514e09828f86917715f686e5dc9681446903
Bug 1232005 - TaskCluster's Gecko decision task generate a file with all the possible jobs that could be scheduled. r=garndt
(Reporter)

Comment 15

2 years ago
Removing NI.
Flags: needinfo?(armenzg)

Comment 16

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/a1c5514e0982
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox49: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
(Reporter)

Comment 17

2 years ago
We should have the artifact being produced for all repositories.
Right now, I don't see anywhere for m-c [1][2], however, I see it working for try [3]

My apologies I should have noticed this earlier.

Currently, we schedule *all* TC jobs on every other repository except try.
However, in the future, we will need this as A) coalescing will be in place and B) we will also have jobs scheduled based on the changes in a push (e.g. we won't trigger mobile jobs if the changes only affect desktop).

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=gecko&selectedJob=3866852&revision=c3f5e6079284
[2] https://tools.taskcluster.net/task-inspector/#C4UJf0PhSleAkNDckZENvg/0
[3] https://tools.taskcluster.net/task-inspector/#CRj-D3JCQBaTTf2FTH9Eow/0
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 18

2 years ago
Created attachment 8751868 [details] [diff] [review]
alltasks2.patch
Attachment #8751868 - Flags: review?(armenzg)
Comment on attachment 8751868 [details] [diff] [review]
alltasks2.patch

Review of attachment 8751868 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/tasks/decision/branch.yml
@@ +81,5 @@
>              --owner='{{owner}}'
>              --level='{{level}}'
>              --revision-hash='{{revision_hash}}'
>              --extend-graph > /home/worker/artifacts/graph.json
> +            ./mach taskcluster-graph

There is no reason to run this command again.  the graph produced on a branch is already scheduling everything that can be scheduled for that branch.

If you want a stable file name on all branches to look at (like all_tasks.json) then I would just copy /home/worker/artifacts/graph.json to /home/worker/artifacts/all_tasks.json
(Reporter)

Comment 20

2 years ago
Hi garndt, at the moment it will be the same, however, once we have coalescing it won't be the same.
OK to proceed taking this into account?

On another note, is there a better way to compare changes like this? [1] I thought dustin might have something to help with this.

[1]
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph --pushlog-id=30251 --project=mozilla-central '--message= ' --owner=cbook@mozilla.com --level=3 --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
cannot query vcs info because vcs info not provided
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u all' --owner=cbook@mozilla.com --level=3 --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
cannot query vcs info because vcs info not provided
armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2
(Assignee)

Comment 21

2 years ago
Created attachment 8752173 [details] [diff] [review]
alltasks2.patch

Is this better now?
Attachment #8751868 - Attachment is obsolete: true
Attachment #8751868 - Flags: review?(armenzg)
Attachment #8752173 - Flags: review?(garndt)
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #20)
> Hi garndt, at the moment it will be the same, however, once we have
> coalescing it won't be the same.
> OK to proceed taking this into account?
> 
> On another note, is there a better way to compare changes like this? [1] I
> thought dustin might have something to help with this.
> 
> [1]
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message= '
> --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u
> all' --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2

with the existing graph command, appending "--dry-run" but even then it's not the easiest.  I think with the new graph commands it will be *much* simpler to do these things and I believe he has thought about having a mach command specifically for showing a diff of before and after of a graph.
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #20)
> Hi garndt, at the moment it will be the same, however, once we have
> coalescing it won't be the same.
> OK to proceed taking this into account?
> 
> On another note, is there a better way to compare changes like this? [1] I
> thought dustin might have something to help with this.
> 
> [1]
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message= '
> --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 1
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ ./mach taskcluster-graph
> --pushlog-id=30251 --project=mozilla-central '--message=try: -b do -p all -u
> all' --owner=cbook@mozilla.com --level=3
> --revision-hash=bc35ae7880e914a44f2549b39a084312f6107744 --extend-graph > 2
> cannot query vcs info because vcs info not provided
> armenzg@armenzg-thinkpad:~/repos/mozilla-central$ vimdiff 1 2

re: coalescing
I believe (and I could be wrong) that coalescing in taskcluster will still result in the decision graph scheduling everything, but a service will have a worker coalesce tasks after they have been scheduled.
(In reply to Kalpesh Krishna [:martianwars] from comment #21)
> Created attachment 8752173 [details] [diff] [review]
> alltasks2.patch
> 
> Is this better now?

I think it's ok, but I'll leave it up to armen to see if this fits into what you two are attempting to accomplish.
(Reporter)

Comment 25

2 years ago
Hey Jake, how does coalescing work?
Do all tasks get scheduled with coalescer but a service will have the worker coalesce the tasks?
In other words, would graph.json be mainly the same (when it comes to noumber of tasks) with and without coalescer?

Thanks!

(In reply to Greg Arndt [:garndt] from comment #23)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #20)
> > Hi garndt, at the moment it will be the same, however, once we have
> > coalescing it won't be the same.
> > 
> re: coalescing
> I believe (and I could be wrong) that coalescing in taskcluster will still
> result in the decision graph scheduling everything, but a service will have
> a worker coalesce tasks after they have been scheduled.
Flags: needinfo?(jwatkins)
(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #25)
> Hey Jake, how does coalescing work?
> Do all tasks get scheduled with coalescer but a service will have the worker
> coalesce the tasks?
> In other words, would graph.json be mainly the same (when it comes to
> noumber of tasks) with and without coalescer?
> 
> Thanks!

:garndt is correct.  Right now (although not enabled in prod just yet), the coalesce service simply builds lists of tasks (by listening to pulse) that can be coalesced and serves that list to a requesting worker.  The worker does the coalescing by claiming the task[0] and resolving tasks[0+n] as completed with exception.  Obviously, task[0] is the more recent of all the tasks.  So the graph will not change.
Flags: needinfo?(jwatkins)
(Reporter)

Comment 27

2 years ago
Comment on attachment 8752173 [details] [diff] [review]
alltasks2.patch

Review of attachment 8752173 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks dividehex!

In that case, we can land this.

@martianwars: does this work for you? do you want me to land it for you?
Attachment #8752173 - Flags: review?(garndt) → review+
(Assignee)

Comment 28

2 years ago
armenzg, works for me, let's land this :D
(Reporter)

Comment 29

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/dd29998a7d44c0caa79881afe494024890b91676
Bug 1232005 - Generating all_tasks.json in gecko decision task for all branches. DONTBUILD. CLOSED TREE. r=armenzg
It appears that there might be an '&&' missed after generating the graph and copying it.  This would result in an error similar to here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e0c637d330f&selectedJob=20906139
(Assignee)

Comment 31

2 years ago
Created attachment 8753051 [details] [diff] [review]
alltasks2.patch

Sorry for the really silly mistake
Attachment #8752173 - Attachment is obsolete: true
Attachment #8753051 - Flags: review?(garndt)
(Assignee)

Comment 32

2 years ago
Created attachment 8753055 [details] [diff] [review]
alltasks2.patch

sorry again :(
Attachment #8753051 - Attachment is obsolete: true
Attachment #8753051 - Flags: review?(garndt)
Attachment #8753055 - Flags: review?(garndt)
(Reporter)

Comment 33

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/4dab7153d8470a6f0758b8c9e8cd46515b9f3d8d
Bug 1232005 - Fix typo in dd29998a7d44. CLOSED TREE. r=garndt
Comment on attachment 8753055 [details] [diff] [review]
alltasks2.patch

Review of attachment 8753055 [details] [diff] [review]:
-----------------------------------------------------------------

r+ but there's nothing to do with this patch.  I believe Armen pushed a change already.
Attachment #8753055 - Flags: review?(garndt) → review+

Comment 35

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/dd29998a7d44
https://hg.mozilla.org/mozilla-central/rev/4dab7153d847
Status: REOPENED → RESOLVED
Last Resolved: 2 years agoa year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.