Closed Bug 1494415 Opened Last year Closed Last year

taskgraph schema defaults not recognized

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(firefox-esr60 fixed, firefox64 fixed)

RESOLVED FIXED
mozilla64
Tracking Status
firefox-esr60 --- fixed
firefox64 --- fixed

People

(Reporter: rjl, Assigned: tomprince)

Details

Attachments

(1 file)

Adding a new field to a TaskGraph Schema along the lines of:

from taskgraph.util.schema import Schema
from voluptuous import Required
from taskgraph.config import GraphConfig
schema = Schema({
    Required('key1'): basestring,
    Required('key2', default='value2'): basestring
})
# Skipped the part where my_config is loaded from a yaml file
my_config={'key1': 'value1'}

g = GraphConfig(config=my_config, root_dir='taskcluster/ci')
>>> g
GraphConfig(_config={'key1': 'value1'}, root_dir='taskcluster/ci')

g2 = GraphConfig(config=schema(my_config), root_dir='taskcluster/ci')
>>> g2
GraphConfig(_config={'key2': 'value2', 'key1': 'value1'}, root_dir='taskcluster/ci')

The expected result is g2, where the loaded yaml data is passed through schema before being applied to GraphConfig.

The root of the problem occurs at
https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/util/schema.py#26
in validate_schema where the return value of schema(obj) is dropped. Returning the value instead would give us a copy of obj with any defaults applied.

In the case of loading a GraphConfig, changes need to be make to config.py:load_graph_config to make use of the returned object.

It looks like defaults are applied in code during the transform process for various tasks, but no such mechanism is in place for the overall graph config.
The key reason we removed use of schema defaults (noting that they still exist in the schema definitions.. ugh) is that when `taskgraph.fast`, we skip validation -- which if we were using the result of applying defaults would mean that behavior changes when we skip validation.

That's certainly less important for the GraphConfig schema, so one quick fix here may be to validate the schema directly rather than using the taskgraph.util.validate_schema helper function.
I thought there might be a reason. I'll write a patch for GraphConfig.
We don't apply schema defaults, since when `taskgraph.fast`, we skip validation
-- which if we were using the result of applying defaults would mean that
behavior changes when we skip validation.
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/24a3c513ce5d
[taskgraph] Remove ignored default from graph config; r=dustin
https://hg.mozilla.org/mozilla-central/rev/24a3c513ce5d
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Assignee: nobody → mozilla
You need to log in before you can comment on or make changes to this bug.