1634679 - Output list of manifests selected by shadow schedulers

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Marco Castelluccio [:marco]]

Needed for showing manifest-level data on the dashboard (bug 1603467 / https://github.com/mozilla/cia-tasks/issues/1).

Comment 1

[Andrew Halberstadt [:ahal]]

Do we want all manifests including the collateral damage ones? If not this will be a bit trickier.

Comment 2

[Marco Castelluccio [:marco]]

I just wanted the list of actually selected manifests, not the collateral damage ones, so I can measure how the schedulers will be doing when we will have manifest-level scheduling.
I can just evaluate on historical data until this is more easily feasible though.

Comment 3

[Marco Castelluccio [:marco]]

Could we simply make the shadow schedulers output the task graph as JSON instead of as a list of labels, and then let consumers choose how to use the data?

E.g. a consumer could:

1. Load the task graph as JSON;
2. List all manifests from manifest-level tasks by using MOZHARNESS_TEST_PATHS;
3. List all non-manifest-level tasks (by filtering out tasks that have MOZHARNESS_TEST_PATHS).

Comment 4

[Andrew Halberstadt [:ahal]]

Yeah, I like that idea.

For now we could just paste the optimized taskgraph as an artifact (would require minimal changes to the shadow-scheduler task). In the future, we could re-do the shadow-schedulers such that they get invoked via decision.py and thus upload all the same artifacts that a normal decision task would. I'm not sure if that's important or not though.

Comment 5

[Andrew Halberstadt [:ahal]]

This will put some extra strain on the ETL, and we'll need to coordinate landing it with Kyle.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1634679 - [ci] Output full taskgraph JSON from shadow-schedulers, r?marco

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/82fcb84ee78a
[ci] Output full taskgraph JSON from shadow-schedulers, r=marco

Comment 8

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/82fcb84ee78a