1841637 - Run taskgraph schema validation in a linting task and rip it out from the decision task

Status: NEW

(Firefox Build System :: Task Configuration, task)

Description

[Marco Castelluccio [:marco]]

jcristau found out schema validation is a big part of decision task runtime. We could move it to a linting task and keep the decision task lean.
This doesn't save any cost, actually it slightly increases them, but allows us to start following tasks much more quickly so it is a win.

Comment 1

[Julien Cristau [:jcristau]]

I'd say it'd only makes sense to skip validation 1) on try, and 2) if nothing under taskcluster/ changes. If either of those isn't true then we're better off finding issues during decision.

Comment 2

[Andrew Halberstadt [:ahal]]

I think there is still benefit to speeding up decision tasks on autoland / central, but agree that try is where most of the pain is felt. So I can see the argument for still validating in-decision for L3 branches..

Another approach could be to always validate in a lint task everywhere, but then additionally validate in the Decision task if files under /taskcluster are modified (regardless of branch). That way we get the speedups on autoland/central as well, and still fail fast when bustage is most likely to happen. Though this is at the cost of sometimes validating twice.

It's worth noting that the lint task could download the taskgraph from the Decision task, so we don't necessarily need to re-generate it a second time. Though re-generating it would be the easiest implementation wise.

Comment 3

[Andrew Halberstadt [:ahal]]

It's worth noting that the lint task could download the taskgraph from the Decision task, so we don't necessarily need to re-generate it a second time. Though re-generating it would be the easiest implementation wise.

This is not true. I think I was conflating schema validation with the post generation verifications. The latter can download artifacts from the Decision task, but the former runs at many intermediate steps throughout graph generation, so there's not any way of getting around generating the full graph.

Comment 4

[Johan Lorenzo [:jlorenzo] - On PTO - Back on July 28th]

Linking this bug to bug 1652123 which also showed voluptuous was known to be slow. That bug was closed in favor of taskgraph#8