1641058 - tasks that only exist because of another that is later optimized away should not be scheduled

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[[e-mail] (:bhearsum)]

While discussing other things, it came up that if task A is included in a target graph only because of task B, and task B is later optimized away, task A will still be scheduled for no reason. It's really difficult to know exactly how often this is happening, but it appears to be pretty common on try and autoland.

Comment 1

[[e-mail] (:bhearsum)]

From Matrix:
bhearsum: If we pass down the set of tasks that we to generate the target graph (i.e the tasks from https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/generator.py#322 but not the closure thereof) and feed it down to https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/optimize/init.py#121 (via https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/generator.py#340 ) we could unconditionally optimize all tasks that aren't in that set.
I'd definitely worry about unexpected side-effects of doing that, though.
aki
we could maybe log what would have been optimized out, and watch that list over time ?
tom.prince
Sure, lots of stuff we could do; I haven't though through what might be impacted. Mostly I was suggesting do lots of testing of the results before landing.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: Bug 1641058: tasks that only exist because of another that is later optimized away should not be scheduled

Comment 3

[[e-mail] (:bhearsum)]

Cost query for this work: https://console.cloud.google.com/bigquery?sq=586736774393:39ad408d46314561b2a1bf718d6e8a92

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6a42da21836e
tasks that only exist because of another that is later optimized away should not be scheduled r=tomprince

Comment 5

[Narcis Beleuzu [:NarcisB]]

Backed out for breaking action tasks like retriggers

Backout link: https://hg.mozilla.org/integration/autoland/rev/5850f0fdfab1d2e8023bc727afec9603efaa939c
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308368211&repo=autoland&lineNumber=43

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d445091a0585
tasks that only exist because of another that is later optimized away should not be scheduled r=tomprince

Comment 7

[[e-mail] (:bhearsum)]

Attachment: Bug 1641058: fix unit test failures from remove_tasks interface change

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/10b03f2e2a93
fix unit test failures from remove_tasks interface change. a=fix. CLOSED TREE

Comment 9

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/d445091a0585
https://hg.mozilla.org/mozilla-central/rev/10b03f2e2a93