1422133 - Generate runnable-jobs.json metadata as part of the Gecko decision task run

Status: RESOLVED FIXED

(Taskcluster :: General, enhancement)

Description

[Armen [:armenzg]]

The runnable jobs API on Treeherder started timing out quite frequently.
This is due that it downloads the full-task-graph.json artifact from a Gecko decision task.

This file has grown to be massive (over 30MB) and that takes too long to fetch. The API timesout after 20 seconds.

The data we need from such artifact is minimal. We can generate a file with just the minimum amount of data needed.

This code change adds logic to generate a runnable-jobs.json file after the full-task-graph.json artifact is generated.

Comment 1

[Armen [:armenzg]]

Attachment: Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

The runnable jobs API on Treeherder started timing out quite frequently.
This is due that it downloads the full-task-graph.json artifact from a Gecko decision task.

This file has grown to be massive (over 30MB) and that takes too long to fetch. The API timesout after 20 seconds.

The data we need from such artifact is minimal. We can generate a file with just the minimum amount of data needed.

This code change adds logic to generate a runnable-jobs.json file after the full-task-graph.json artifact is generated.

Review commit: https://reviewboard.mozilla.org/r/204618/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/204618/

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I don't think NPOTB is appropriate for this patch -- it could easily cause failures in automation.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210212

::: taskcluster/docs/runnable_jobs.rst:1
(Diff revision 1)
> +Runnable jobs

Thanks for documenting!  But without a link from another file, this will not appear.

Rather than being its own file, I think this would do well as a section in https://firefox-source-docs.mozilla.org/taskcluster/taskcluster/taskgraph.html near the actions section (which also describes an extra artifact).

::: taskcluster/taskgraph/decision.py:98
(Diff revision 1)
> +    for label, node in full_task_json.iteritems():
> +        if not ('extra' in node['task'] and 'treeherder' in node['task']['extra']):
> +            continue
> +        runnable_jobs[label] = {
> +            'treeherder': node['task']['extra']['treeherder'],
> +            'metadata': {

This seems an unusual format for what appears to be four useful pieces of data (label, symbol, platform, and description).  Why not flatten it to just include that data?  That will save quite a bit of space and keep the file shorter.

::: taskcluster/taskgraph/decision.py:100
(Diff revision 1)
> +            continue
> +        runnable_jobs[label] = {
> +            'treeherder': node['task']['extra']['treeherder'],
> +            'metadata': {
> +                'description': node['task']['metadata']['description'],
> +                'name': node['task']['metadata']['name']

isn't name always the same as label?

::: taskcluster/taskgraph/test/test_util_runnable_jobs.py:11
(Diff revision 1)
> +from taskgraph.decision import full_task_graph_to_runnable_jobs
> +from taskgraph.task import Task
> +from mozunit import main
> +
> +
> +class TestRunnableJobs(MorphTestCase):

Why is this a MorphTestCase?

Comment 4

[Armen [:armenzg]]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210234

::: taskcluster/taskgraph/test/test_util_runnable_jobs.py:11
(Diff revision 1)
> +from taskgraph.decision import full_task_graph_to_runnable_jobs
> +from taskgraph.task import Task
> +from mozunit import main
> +
> +
> +class TestRunnableJobs(MorphTestCase):

To have access to self.make_taskgraph().

Is that a good enough reason? What do you suggest?

Comment 5

[Armen [:armenzg]]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210212

> isn't name always the same as label?

When I first implemented it I was not sure if this would be guaranteed.
If that is the case, why do we have 'name' specified within 'metadata'?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Armen [:armenzg] from comment #4)
> To have access to self.make_taskgraph().
> 
> Is that a good enough reason? What do you suggest?

No :)

Implement something similar, even copy-paste into this test script.  There's no need to create dependencies where none exist.

> When I first implemented it I was not sure if this would be guaranteed.
> If that is the case, why do we have 'name' specified within 'metadata'?

The metadata is given to the queue, and allows us to see the task label there after taskgraph generation is complete. `task.label` is used internally as the label.

Comment 7

[Armen [:armenzg]]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/204618/diff/1-2/

Comment 8

[Armen [:armenzg]]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/204618/diff/2-3/

Comment 9

[Armen [:armenzg]]

dustin: emorley suggested I offer this file compressed. Any suggested approach to doing so?
Just use zipfile? or anything else?

Comment 10

[Armen [:armenzg]]

gzipped that is

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, I think so, and write it with name foo.json.gz.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210762

Looks good.  I'll be happy to re-r? an update to use .gz.

Comment 13

[Armen [:armenzg]]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/204618/diff/3-4/

Comment 14

[Armen [:armenzg]]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

The interdiff is minimal. Thank you!

TH run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=de14f9b796c8f83e10a350b1567f6c1faa7c7c03
Artifact: https://queue.taskcluster.net/v1/task/asDdxxp4TuCSwhF3uYEtKg/runs/0/artifacts/public/runnable-jobs.json.gz

The file generated is less than 100kb.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

https://reviewboard.mozilla.org/r/204618/#review210774

Comment 16

[Armen [:armenzg]]

Comment on attachment 8933674 [details]
Bug 1422133 - Generate runnable-jobs.json.gz file as part of the Gecko decision task run.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/204618/diff/4-5/

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3881a65bd690
Generate runnable-jobs.json.gz file as part of the Gecko decision task run. r=dustin

Comment 18

[Ryan VanderMeulen [:RyanVM]]

We should land this on Beta as well since there's still 6 weeks left in this current cycle.

Comment 19

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/3881a65bd690

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/60ab7fff591c