Closed Bug 1422133 Opened 7 years ago Closed 7 years ago

Generate runnable-jobs.json metadata as part of the Gecko decision task run

Categories

(Taskcluster :: General, enhancement)

Product:
Taskcluster
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla59

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

Attachments

(1 file)

Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.
59 bytes, text/x-review-board-request
dustin
: review+
Details 
The runnable jobs API on Treeherder started timing out quite frequently.
This is due that it downloads the full-task-graph.json artifact from a Gecko decision task.

This file has grown to be massive (over 30MB) and that takes too long to fetch. The API timesout after 20 seconds.

The data we need from such artifact is minimal. We can generate a file with just the minimum amount of data needed.

This code change adds logic to generate a runnable-jobs.json file after the full-task-graph.json artifact is generated.
Attachment #8933674 - Flags: review?(dustin)
I don't think NPOTB is appropriate for this patch -- it could easily cause failures in automation.
Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210212

::: taskcluster/docs/runnable_jobs.rst:1
(Diff revision 1)
> +Runnable jobs

Thanks for documenting!  But without a link from another file, this will not appear.

Rather than being its own file, I think this would do well as a section in https://firefox-source-docs.mozilla.org/taskcluster/taskcluster/taskgraph.html near the actions section (which also describes an extra artifact).

::: taskcluster/taskgraph/decision.py:98
(Diff revision 1)
> +    for label, node in full_task_json.iteritems():
> +        if not ('extra' in node['task'] and 'treeherder' in node['task']['extra']):
> +            continue
> +        runnable_jobs[label] = {
> +            'treeherder': node['task']['extra']['treeherder'],
> +            'metadata': {

This seems an unusual format for what appears to be four useful pieces of data (label, symbol, platform, and description).  Why not flatten it to just include that data?  That will save quite a bit of space and keep the file shorter.

::: taskcluster/taskgraph/decision.py:100
(Diff revision 1)
> +            continue
> +        runnable_jobs[label] = {
> +            'treeherder': node['task']['extra']['treeherder'],
> +            'metadata': {
> +                'description': node['task']['metadata']['description'],
> +                'name': node['task']['metadata']['name']

isn't name always the same as label?

::: taskcluster/taskgraph/test/test_util_runnable_jobs.py:11
(Diff revision 1)
> +from taskgraph.decision import full_task_graph_to_runnable_jobs
> +from taskgraph.task import Task
> +from mozunit import main
> +
> +
> +class TestRunnableJobs(MorphTestCase):

Why is this a MorphTestCase?
Attachment #8933674 - Flags: review?(dustin) → review-
Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210234

::: taskcluster/taskgraph/test/test_util_runnable_jobs.py:11
(Diff revision 1)
> +from taskgraph.decision import full_task_graph_to_runnable_jobs
> +from taskgraph.task import Task
> +from mozunit import main
> +
> +
> +class TestRunnableJobs(MorphTestCase):

To have access to self.make_taskgraph().

Is that a good enough reason? What do you suggest?
Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210212

> isn't name always the same as label?

When I first implemented it I was not sure if this would be guaranteed.
If that is the case, why do we have 'name' specified within 'metadata'?
(In reply to Armen [:armenzg] from comment #4)
> To have access to self.make_taskgraph().
> 
> Is that a good enough reason? What do you suggest?

No :)

Implement something similar, even copy-paste into this test script.  There's no need to create dependencies where none exist.

> When I first implemented it I was not sure if this would be guaranteed.
> If that is the case, why do we have 'name' specified within 'metadata'?

The metadata is given to the queue, and allows us to see the task label there after taskgraph generation is complete. `task.label` is used internally as the label.
dustin: emorley suggested I offer this file compressed. Any suggested approach to doing so?
Just use zipfile? or anything else?
gzipped that is
Yeah, I think so, and write it with name foo.json.gz.
Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210762

Looks good.  I'll be happy to re-r? an update to use .gz.
Attachment #8933674 - Flags: review?(dustin) → review+
Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

The interdiff is minimal. Thank you!

TH run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=de14f9b796c8f83e10a350b1567f6c1faa7c7c03
Artifact: https://queue.taskcluster.net/v1/task/asDdxxp4TuCSwhF3uYEtKg/runs/0/artifacts/public/runnable-jobs.json.gz

The file generated is less than 100kb.
Attachment #8933674 - Flags: review+ → review?(dustin)
Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210774
Attachment #8933674 - Flags: review?(dustin) → review+
Pushed by armenzg@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3881a65bd690
Generate runnable-jobs.json.gz file as part of the Gecko decision task run. r=dustin
We should land this on Beta as well since there's still 6 weeks left in this current cycle.
Whiteboard: [checkin-needed-beta]
https://hg.mozilla.org/mozilla-central/rev/3881a65bd690
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
Depends on: 1423215
https://hg.mozilla.org/releases/mozilla-beta/rev/60ab7fff591c
Whiteboard: [checkin-needed-beta]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: