Closed Bug 1379163 Opened 3 years ago Closed 3 years ago

Implement trigger missing jobs feature

Categories

(Taskcluster :: General, enhancement)

enhancement
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla56

People

(Reporter: armenzg, Assigned: dustin)

References

Details

Attachments

(5 files)

There's a Treeherder action feature (pulse_actions + mozci) which is used by sheriffs for merging commits around. Trigger missing jobs analyzes which jobs have not been scheduled on a push and schedules them.

I'm happy to make this my deliverable for this quarter. I will need just a bit of direction as to what code to touch.

Any thoughts on how to implement this?
Is an action task still a good way for implementing it?
Blocks: 1080265
Blocks: 1379172
Please implement in actions.json.  I'm working on fleshing out support for that in Treeherder now (it's there but sort of limited at the moment).
I believe there's a typo on the documentation saying that tasks should be defined as taskcluster/taskgraph/actions/my-action.py instead of taskcluster/actions/my-action.py.

What command do I use to generate an actions.json file locally?
The one a decision task executes [1]?

Is there also a command to test a created task directly from my local machine? I could not find it on the documentation.

Also, do you have any suggestion on how or where to trasverse the task graph to determine what was scheduled and what was not?
Is this information something I can discover through some of the artifacts of the gecko decision task?

Thanks in advance.

[1]
./mach --log-no-times taskgraph decision --pushlog-id=32145 --pushdate=1499443022 --project=mozilla-central '--message= ' --owner=ffxbld@noreply.mozilla.org --level=3 --base-repository=https://hg.mozilla.org/mozilla-central --head-repository=https://hg.mozilla.org/mozilla-central/ --head-ref=5eeee16f1659ca1f7c4e28e5834d051eb060eafc --head-rev=5eeee16f1659ca1f7c4e28e5834d051eb060eafc
Depends on: 1335920
Assignee: nobody → armenzg
bstack will be looking at older style actions tasks this week and describe what we will need for this.

The following is some of what we discussed:

armenzg said:
1) create a mach command that can determine what is missing and schedule them
2) add an action py file under taskcluster/actions that would call that mach command
3) modify TH to use actions.json and show push/resulset level actions

bstack said: "I'm not sure we need it to be a mach thing, but it probably isn't an issue either way"
Treeherder support of actions.json for job-level actions was added here:
https://github.com/mozilla/treeherder/commit/a45f2d3c48c0b6e79fe26609d1599558ad91c44d
Assignee: armenzg → bstack
Blocks: 1370343
I'll see if I can work on this.
Assignee: bstack → dustin
Note that this assumes that any test dependencies were also built.  We don't currently optimize away builds when tests are optimized away, so I don't think that will be a problem.  If it is, it's just a little more logic to add to expand `to_run` with any tasks that are dependencies of other tasks in to_run and that have not themselves been run.
[task 2017-07-21T13:18:51.519609Z] ImportError: No module named util
[task 2017-07-21T13:18:51.519628Z] 
[task 2017-07-21T13:18:51.519679Z]   File "/home/worker/checkouts/gecko/taskcluster/mach_commands.py", line 150, in taskgraph_decision
[task 2017-07-21T13:18:51.519716Z]     import taskgraph.decision
I forgot, mach gives exceptions backward.  Reversing, and filtering out the custom import it makes a lot more sense:

[task 2017-07-21T13:18:51.519609Z] ImportError: No module named util
[task 2017-07-21T13:18:51.519628Z] 
[task 2017-07-21T13:18:51.520442Z]   File "/home/worker/checkouts/gecko/taskcluster/actions/run_missing_tests.py", line 13, in <module>
[task 2017-07-21T13:18:51.520477Z]     from actions.util import create_task
[task 2017-07-21T13:18:51.520258Z]   File "/home/worker/checkouts/gecko/taskcluster/actions/registry.py", line 317, in <module>
[task 2017-07-21T13:18:51.520291Z]     __import__('actions.' + f[:-3])
[task 2017-07-21T13:18:51.520073Z]   File "/home/worker/checkouts/gecko/taskcluster/actions/__init__.py", line 9, in <module>
[task 2017-07-21T13:18:51.520112Z]     from .registry import (
[task 2017-07-21T13:18:51.519869Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 20, in <module>
[task 2017-07-21T13:18:51.519908Z]     from actions import render_actions_json
[task 2017-07-21T13:18:51.519679Z]   File "/home/worker/checkouts/gecko/taskcluster/mach_commands.py", line 150, in taskgraph_decision
[task 2017-07-21T13:18:51.519716Z]     import taskgraph.decision
Comment on attachment 8888490 [details]
Bug 1379163: Implement the "Run Missing Tests" action

https://reviewboard.mozilla.org/r/159452/#review165170

lgtm. Thanks Dustin!
Attachment #8888490 - Flags: review?(armenzg) → review+
sheriffs: FYI the trigger missing jobs will be slightly different than on Buildbot (at the implementation level) as TaskCluster always creates builds, thus, it will be looking at triggering test tasks that were optimized (skipped). This I believe should work for you.

dustin: I was thinking; can you output the labels of the tasks that get scheduled? or will it be easy to figure out which tasks gets scheduled through other means? (e.g. looking at a task group or something similar). I want it easy to inspect a posteriori.
Yeah, I can add that, good thinking.  Also, the exception above was because bug 1382707 hadn't landed yet.  And note that its bugfix followup hasn't landed yet either, so at the moment actions don't work.
Ah, I see that it already does that (create_task has `logger.debug("Creating task with taskId {} for {}".format(task_id, label))`
https://treeherder.mozilla.org/#/jobs?repo=try&revision=cc6bf85e3e8f690c37ff5961f0a7cbf55792ffb3&selectedJob=116433890

https://public-artifacts.taskcluster.net/fTybOeR9RvusrNOmQHIxSQ/0/public/logs/live_backing.log is not very exciting, but you get the idea!  From my local runs against m-c, this does in fact schedule tests when there are tests to schedule.  To trigger this action:
 - select a task in treeherder
 - ... -> custom action
 - choose "run missing tests"
 - type "null" in the JSON payload form
 - click "Trigger"
Comment on attachment 8888815 [details]
Bug 1379163: defer loading actions until needed;

https://reviewboard.mozilla.org/r/159850/#review165280

Nice :)
Attachment #8888815 - Flags: review?(jopsen) → review+
Comment on attachment 8888488 [details]
Bug 1379163: add missing license headers;

https://reviewboard.mozilla.org/r/159448/#review165298
Attachment #8888488 - Flags: review?(bstack) → review+
Comment on attachment 8888489 [details]
Bug 1379163: make parameterization functions into utilities;

https://reviewboard.mozilla.org/r/159450/#review165308
Attachment #8888489 - Flags: review?(bstack) → review+
Comment on attachment 8888490 [details]
Bug 1379163: Implement the "Run Missing Tests" action

https://reviewboard.mozilla.org/r/159452/#review165380

Otherwise this makes a lot of sense! Let's talk Monday about which bits should be moved to util!

::: taskcluster/actions/run_missing_tests.py:46
(Diff revision 3)
> +def run_missing_tests(parameters, input, task_group_id, task_id, task):
> +    decision_task_id = find_decision_task(parameters)
> +
> +    full_task_graph = get_artifact(decision_task_id, "public/full-task-graph.json")
> +    _, full_task_graph = TaskGraph.from_json(full_task_graph)
> +    task_graph = get_artifact(decision_task_id, "public/task-graph.json")

This is never used, right? I only see `full_task_graph` from here (and the line below it) on out.
Attachment #8888490 - Flags: review?(bstack) → review+
Yeah, autoland will make our collaboration a little difficult.  We could collaborate temporarily in a user repo..  Let me know when you want to talk.
Flags: needinfo?(bstack)
Flags: needinfo?(bstack)
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/963b59bd61a3
add missing license headers; r=bstack
https://hg.mozilla.org/integration/autoland/rev/2e143d567065
make parameterization functions into utilities; r=bstack
https://hg.mozilla.org/integration/autoland/rev/aff22b63df7a
Implement the "Run Missing Tests" action; r=armenzg,bstack
https://hg.mozilla.org/integration/autoland/rev/03896a3c7a68
defer loading actions until needed; r=jonasfj
Follow ups are bug 1383066 and now bug 1388407.
You need to log in before you can comment on or make changes to this bug.