Replace voluptuous for schema validation in taskgraph.
Categories
(Firefox Build System :: Task Configuration, enhancement)
Tracking
(firefox149 affected)
| Tracking | Status | |
|---|---|---|
| firefox149 | --- | affected |
People
(Reporter: tomprince, Assigned: abhishekmadan)
References
(Blocks 2 open bugs)
Details
(Keywords: leave-open)
Attachments
(58 files, 11 obsolete files)
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
47 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
Bug 1652123 - Converted BeetmoverSourceChecksumsDescriptionSchema from voluptuous to msgspec. r=ahal
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review |
I've wanted to replace voluptuous for a while, due to https://github.com/alecthomas/voluptuous/issues/287.
While working on Bug 1648591, and rewritting a schema from taskgraph using json-schema for another project (in order to avoid adding voluptuous as a dependency). I realized that it would not be difficult to implement enough of voluptuous' API as a wrapper around jsonschema, and implemented a prototype.
Looking at Bug 1652086, I was curious what kind of performance impact it has.
On my machine, when skipping schema validation, my prototype is ~45s faster, and when not skipping validation, it is ~70s faster.
| Reporter | ||
Comment 1•5 years ago
|
||
Correction, the json-schema version is validating in both cases, the comparison is total time with --fast vs without --fast, which disables schema validation for voluptuous (among other things).
| Reporter | ||
Comment 2•5 years ago
|
||
| Reporter | ||
Comment 3•5 years ago
|
||
Comment 4•4 years ago
|
||
Note I replaced voluptuous with validx in another project, it looked like this:
https://github.com/mozilla/mozci/commit/ceab0279e7e0b0771c8d1ee3170ad3e0789971dc
It claims to be 11x faster than jsonschema (but only 3x faster than voluptuous), so I guess their benchmark is quite different than our use case.
While that project was much simpler, converting the format wasn't that big a deal. We'd need to re-implement our custom validators like optionally_keyed_by, but it should be do-able.
My preference would be to not bother with a wrapping layer to preserve the syntax, and lean on scripting to handle the conversion.
Comment 5•4 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #4)
My preference would be to not bother with a wrapping layer to preserve the syntax, and lean on scripting to handle the conversion.
I think the tricky bit here is we may have schemas defined in-tree, e.g. in Fenix or in A-C, and if we change the format in taskgraph, they can't upgrade to a newer taskgraph unless they also change their schema definitions. Or we'd need to go and find all the consumers of standalone taskgraph and provide patches to use the new schema format. Neither are hard blockers to going this route; just something to consider.
Comment 6•3 years ago
|
||
| Assignee | ||
Updated•6 months ago
|
| Assignee | ||
Comment 8•6 months ago
|
||
Updated•6 months ago
|
| Assignee | ||
Comment 9•5 months ago
|
||
Updated•5 months ago
|
Updated•5 months ago
|
Updated•4 months ago
|
Comment 10•1 month ago
|
||
Comment 11•1 month ago
|
||
Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/c73b25c97961
Port to Thunderbird - Part 1: Refactored voluptuous schemas to now be called LegacySchema. rs=bustage-fix DONTBUILD
Updated•1 month ago
|
Comment 12•1 month ago
|
||
| bugherder | ||
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
| Assignee | ||
Comment 13•1 month ago
|
||
| Assignee | ||
Comment 14•1 month ago
|
||
| Assignee | ||
Comment 15•1 month ago
|
||
| Assignee | ||
Comment 16•1 month ago
|
||
| Assignee | ||
Comment 17•1 month ago
|
||
| Assignee | ||
Comment 18•1 month ago
|
||
| Assignee | ||
Comment 19•1 month ago
|
||
| Assignee | ||
Comment 20•1 month ago
|
||
| Assignee | ||
Comment 21•1 month ago
|
||
| Assignee | ||
Comment 22•1 month ago
|
||
| Assignee | ||
Comment 23•1 month ago
|
||
| Assignee | ||
Comment 24•1 month ago
|
||
| Assignee | ||
Comment 25•1 month ago
|
||
| Assignee | ||
Comment 26•1 month ago
|
||
Updated•1 month ago
|
Updated•1 month ago
|
| Assignee | ||
Comment 27•24 days ago
|
||
| Assignee | ||
Comment 28•24 days ago
|
||
| Assignee | ||
Comment 29•24 days ago
|
||
| Assignee | ||
Comment 30•24 days ago
|
||
Updated•24 days ago
|
| Assignee | ||
Comment 31•20 days ago
|
||
| Assignee | ||
Comment 32•20 days ago
|
||
| Assignee | ||
Comment 33•20 days ago
|
||
| Assignee | ||
Comment 34•20 days ago
|
||
| Assignee | ||
Comment 35•20 days ago
|
||
Updated•20 days ago
|
Updated•20 days ago
|
Updated•20 days ago
|
Updated•20 days ago
|
Updated•20 days ago
|
| Assignee | ||
Comment 36•20 days ago
|
||
| Assignee | ||
Comment 37•20 days ago
|
||
| Assignee | ||
Comment 38•20 days ago
|
||
| Assignee | ||
Comment 39•20 days ago
|
||
| Assignee | ||
Comment 40•20 days ago
|
||
Updated•20 days ago
|
Updated•20 days ago
|
Updated•20 days ago
|
Updated•20 days ago
|
Comment 41•20 days ago
|
||
Comment 42•20 days ago
|
||
| bugherder | ||
Comment 43•17 days ago
|
||
Comment 44•17 days ago
|
||
Comment 45•17 days ago
|
||
Comment 46•17 days ago
|
||
Comment 47•17 days ago
|
||
Comment 48•17 days ago
|
||
| bugherder | ||
Comment 49•15 days ago
|
||
Updated•15 days ago
|
Comment 50•15 days ago
|
||
| bugherder | ||
Comment 51•14 days ago
|
||
Comment 52•13 days ago
|
||
| bugherder | ||
| Assignee | ||
Comment 53•13 days ago
|
||
| Assignee | ||
Comment 54•13 days ago
|
||
| Assignee | ||
Comment 55•13 days ago
|
||
| Assignee | ||
Comment 56•13 days ago
|
||
| Assignee | ||
Comment 57•13 days ago
|
||
| Assignee | ||
Comment 58•13 days ago
|
||
| Assignee | ||
Comment 59•13 days ago
|
||
| Assignee | ||
Comment 60•13 days ago
|
||
| Assignee | ||
Comment 61•13 days ago
|
||
Comment 62•13 days ago
|
||
Comment 63•13 days ago
|
||
Comment 64•13 days ago
|
||
| bugherder | ||
Comment 65•11 days ago
|
||
Comment 66•11 days ago
|
||
Comment 67•11 days ago
|
||
Comment 68•11 days ago
|
||
Backed out for causing ci failures at test_mozilla_central.py
Backout link
Push with failures
Failure log(s)
Failure line: taskcluster/test/test_mozilla_central.py::test_test_setting TEST-UNEXPECTED-FAIL
| Assignee | ||
Comment 69•10 days ago
|
||
| Assignee | ||
Comment 70•10 days ago
|
||
| Assignee | ||
Comment 71•10 days ago
|
||
| Assignee | ||
Comment 72•10 days ago
|
||
| Assignee | ||
Comment 73•10 days ago
|
||
| Assignee | ||
Comment 74•10 days ago
|
||
| Assignee | ||
Comment 75•10 days ago
|
||
| Assignee | ||
Comment 76•10 days ago
|
||
| Assignee | ||
Comment 77•10 days ago
|
||
| Assignee | ||
Comment 78•10 days ago
|
||
| Assignee | ||
Comment 79•10 days ago
|
||
| Assignee | ||
Comment 80•10 days ago
|
||
| Assignee | ||
Comment 81•10 days ago
|
||
| Assignee | ||
Comment 82•10 days ago
|
||
| Assignee | ||
Comment 83•10 days ago
|
||
| Assignee | ||
Comment 84•10 days ago
|
||
| Assignee | ||
Comment 85•10 days ago
|
||
| Assignee | ||
Comment 86•10 days ago
|
||
| Assignee | ||
Comment 87•10 days ago
|
||
| Assignee | ||
Comment 88•10 days ago
|
||
| Assignee | ||
Comment 89•10 days ago
|
||
| Assignee | ||
Comment 90•10 days ago
|
||
| Assignee | ||
Comment 91•10 days ago
|
||
| Assignee | ||
Comment 92•10 days ago
|
||
| Assignee | ||
Comment 93•10 days ago
|
||
| Assignee | ||
Comment 94•10 days ago
|
||
| Assignee | ||
Comment 95•10 days ago
|
||
| Assignee | ||
Comment 96•10 days ago
|
||
Comment 97•10 days ago
|
||
Comment 98•10 days ago
|
||
Comment 99•10 days ago
|
||
Comment 100•10 days ago
|
||
| bugherder | ||
Comment 101•9 days ago
|
||
Comment 102•9 days ago
|
||
| bugherder | ||
Comment 103•9 days ago
|
||
Comment 104•9 days ago
|
||
Comment 105•9 days ago
|
||
| bugherder | ||
Comment 106•8 days ago
|
||
| bugherder | ||
Comment 107•8 days ago
|
||
| Assignee | ||
Updated•8 days ago
|
Comment 108•8 days ago
|
||
| bugherder | ||
Comment 109•8 days ago
|
||
Comment 110•8 days ago
|
||
Comment 111•8 days ago
|
||
Comment 112•8 days ago
|
||
Comment 113•8 days ago
|
||
| bugherder | ||
Updated•8 days ago
|
Comment 114•7 days ago
|
||
Comment 115•7 days ago
|
||
Comment 116•7 days ago
|
||
Comment 117•7 days ago
|
||
Comment 118•7 days ago
|
||
Comment 119•7 days ago
|
||
Comment 120•7 days ago
|
||
Comment 121•7 days ago
|
||
Comment 122•7 days ago
|
||
Comment 123•7 days ago
|
||
Comment 124•7 days ago
|
||
Comment 125•7 days ago
|
||
Comment 126•7 days ago
|
||
Comment 127•7 days ago
|
||
Comment 128•7 days ago
|
||
Comment 129•7 days ago
|
||
Comment 130•7 days ago
|
||
Comment 131•7 days ago
|
||
Comment 132•7 days ago
|
||
Comment 133•7 days ago
|
||
Comment 134•7 days ago
|
||
Comment 135•7 days ago
|
||
Comment 136•7 days ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/012d1e772900
https://hg.mozilla.org/mozilla-central/rev/8b2efce84742
https://hg.mozilla.org/mozilla-central/rev/291517aa9ff5
https://hg.mozilla.org/mozilla-central/rev/4a31d1f55a36
https://hg.mozilla.org/mozilla-central/rev/0b831b506164
https://hg.mozilla.org/mozilla-central/rev/6692a964a2e3
https://hg.mozilla.org/mozilla-central/rev/c807d25a5425
https://hg.mozilla.org/mozilla-central/rev/d1fb40adbd63
https://hg.mozilla.org/mozilla-central/rev/4328cf4fec96
https://hg.mozilla.org/mozilla-central/rev/53251fc3b5cb
https://hg.mozilla.org/mozilla-central/rev/b5b2c32e1721
https://hg.mozilla.org/mozilla-central/rev/286dad38e22b
https://hg.mozilla.org/mozilla-central/rev/7ef1ddc39314
https://hg.mozilla.org/mozilla-central/rev/f4e910a92bf1
https://hg.mozilla.org/mozilla-central/rev/d6ae7646f90a
https://hg.mozilla.org/mozilla-central/rev/c92e5f3b4314
https://hg.mozilla.org/mozilla-central/rev/34ed79e08785
https://hg.mozilla.org/mozilla-central/rev/c2dd579acbd0
https://hg.mozilla.org/mozilla-central/rev/f722880eff89
https://hg.mozilla.org/mozilla-central/rev/dfb4646deb46
https://hg.mozilla.org/mozilla-central/rev/ea780121b2de
https://hg.mozilla.org/mozilla-central/rev/4ffc0ad4c983
Comment 137•3 days ago
|
||
Comment 138•3 days ago
|
||
| bugherder | ||
Comment 139•3 days ago
|
||
Comment 140•2 days ago
|
||
| bugherder | ||
Comment 141•2 days ago
|
||
| bugherder | ||
Comment 142•1 day ago
|
||
Comment 143•1 day ago
|
||
| bugherder | ||
Description
•