Use json schema validation so issues with payloads posted to the API are easier to debug

RESOLVED WONTFIX

Status

Tree Management
Treeherder: API
P3
normal
RESOLVED WONTFIX
4 years ago
a year ago

People

(Reporter: emorley, Unassigned)

Tracking

Details

(Reporter)

Description

4 years ago
Migrated from:
https://github.com/mozilla/treeherder-service/issues/144

lightsofapollo commented on 15 May:
{
@jonasfj did some great work using json schema validation for the taskcluster ui (https://github.com/taskcluster/task-graph-scheduler/blob/master/schemas/scheduler/v1/create-task-graph-response.json)

I would suggest using json schema to validate at least input which would make treeherder errors much easier to diagnose and friendlier for third party consumers... Currently when I submit some data with missing keys/values (or unexpected types in those values) the errors are often very cryptic (or cause bugs in treeherder which we saw before)
}

jonasfj commented on 15 May:
{
Indeed, this would be nice... If you go ahead an actually do it, feel free to bug me from some code that auto-generates documentation using JSON schemas.
}
Going to bump this again... If we are going to have a public API for treeherder it really needs to be easier to not break it (or at minimum get some better error messages). In 3+ cases now I have hit bugs where I cannot submit data due to internal implementation rules and I need to debug treeherder on my local instance to figure out what is wrong.

The leading underscore bug I hit is a good example of a lack of obvious public data model https://bugzilla.mozilla.org/show_bug.cgi?id=1057909 another example is when submitting a job symbol that is not defined I get a stack trace leading me to read some code which I know means that I submitted some unexpected data (though understanding the TH code base).

I want treeherder to be the place where mozilla sends all of its test and performance data results but we really need to improve the validation here before this is going to be a good experience.

@Jeads: Can we prioritize this ? Jonas from my team knows a lot about this and has written a bunch of auto/doc/auto client gen tools and is happy to help with any questions related to json schema in particular.
Flags: needinfo?(jeads)
(Reporter)

Comment 2

4 years ago
(In reply to James Lal [:lightsofapollo] from comment #1)
> I want treeherder to be the place where mozilla sends all of its test and
> performance data results but we really need to improve the validation here
> before this is going to be a good experience.

Agree this is an important bug to fix before treating treeherder as the single channel for test results at mozilla. 

> @Jeads: Can we prioritize this ? Jonas from my team knows a lot about this
> and has written a bunch of auto/doc/auto client gen tools and is happy to
> help with any questions related to json schema in particular.

Unfortunately at the moment, getting treeherder ready for sheriffs/devs switching to it from TBPL is the first priority & an a-team quarterly goal, with supporting new non-buildbot projects secondary to that,  since at the moment we're going to struggle to meet the q3 goal of switching from TBPL. In addition, until sheriffs are using treeherder, anything submitting to it isn't sheriffed, and we're stuck trying to maintain treeherder regexes/job type support for both TBPL and treeherder, which is an additional burden on sheriffs.
To clarify I agree /w Ed above obviously shipping to users is more important (just so we can drop maintaining so many things) I am asking for priority on this sometime soon after we transition over (TH being primary affects my team more then anything else).
(Reporter)

Comment 4

4 years ago
Sounds good to me :-)
(Reporter)

Comment 5

4 years ago
When we implement this, we can remove the hacky workaround added by bug 1059949:
https://github.com/mozilla/treeherder-ui/commit/0bdb50c7f6a49da7ea680ea85fd0ac546f2fae99
Clearing the needinfo, this is on our list.
Flags: needinfo?(jeads)
(Reporter)

Updated

3 years ago
No longer blocks: 1072676
Component: Treeherder → Treeherder: API
(Reporter)

Updated

3 years ago
Priority: P3 → P5
(Reporter)

Updated

3 years ago
Priority: P5 → P2
Summary: Use json schema validation → Use json schema validation so issues with payloads posted to the API are easier to debug
(Reporter)

Updated

3 years ago
Duplicate of this bug: 1175559
(Reporter)

Updated

3 years ago
Duplicate of this bug: 1174334
(Reporter)

Updated

3 years ago
Blocks: 1196191
Now that we use the JSON schema for results submitted via Mozilla Pulse, will we also update the client now to use the schema?
(Reporter)

Updated

2 years ago
Priority: P2 → P3
(Reporter)

Updated

2 years ago
Blocks: 1270169
(Reporter)

Comment 10

a year ago
Submitting jobs/pushes via the API is now discouraged in favour of using Pulse (bug 1349182), so we're wontfixing API-specific issues.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.