1043338 - Use json schema validation so issues with payloads posted to the API are easier to debug

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: API, defect, P3)

Description

[Ed Morley [:emorley]]

Migrated from:
https://github.com/mozilla/treeherder-service/issues/144

lightsofapollo commented on 15 May:
{
@jonasfj did some great work using json schema validation for the taskcluster ui (https://github.com/taskcluster/task-graph-scheduler/blob/master/schemas/scheduler/v1/create-task-graph-response.json)

I would suggest using json schema to validate at least input which would make treeherder errors much easier to diagnose and friendlier for third party consumers... Currently when I submit some data with missing keys/values (or unexpected types in those values) the errors are often very cryptic (or cause bugs in treeherder which we saw before)
}

jonasfj commented on 15 May:
{
Indeed, this would be nice... If you go ahead an actually do it, feel free to bug me from some code that auto-generates documentation using JSON schemas.
}

Comment 1

[James Lal [:lightsofapollo]]

Going to bump this again... If we are going to have a public API for treeherder it really needs to be easier to not break it (or at minimum get some better error messages). In 3+ cases now I have hit bugs where I cannot submit data due to internal implementation rules and I need to debug treeherder on my local instance to figure out what is wrong.

The leading underscore bug I hit is a good example of a lack of obvious public data model https://bugzilla.mozilla.org/show_bug.cgi?id=1057909 another example is when submitting a job symbol that is not defined I get a stack trace leading me to read some code which I know means that I submitted some unexpected data (though understanding the TH code base).

I want treeherder to be the place where mozilla sends all of its test and performance data results but we really need to improve the validation here before this is going to be a good experience.

@Jeads: Can we prioritize this ? Jonas from my team knows a lot about this and has written a bunch of auto/doc/auto client gen tools and is happy to help with any questions related to json schema in particular.

Comment 2

[Ed Morley [:emorley]]

(In reply to James Lal [:lightsofapollo] from comment #1)
> I want treeherder to be the place where mozilla sends all of its test and
> performance data results but we really need to improve the validation here
> before this is going to be a good experience.

Agree this is an important bug to fix before treating treeherder as the single channel for test results at mozilla. 

> @Jeads: Can we prioritize this ? Jonas from my team knows a lot about this
> and has written a bunch of auto/doc/auto client gen tools and is happy to
> help with any questions related to json schema in particular.

Unfortunately at the moment, getting treeherder ready for sheriffs/devs switching to it from TBPL is the first priority & an a-team quarterly goal, with supporting new non-buildbot projects secondary to that,  since at the moment we're going to struggle to meet the q3 goal of switching from TBPL. In addition, until sheriffs are using treeherder, anything submitting to it isn't sheriffed, and we're stuck trying to maintain treeherder regexes/job type support for both TBPL and treeherder, which is an additional burden on sheriffs.

Comment 3

[James Lal [:lightsofapollo]]

To clarify I agree /w Ed above obviously shipping to users is more important (just so we can drop maintaining so many things) I am asking for priority on this sometime soon after we transition over (TH being primary affects my team more then anything else).

Comment 4

[Ed Morley [:emorley]]

Sounds good to me :-)

Comment 5

[Ed Morley [:emorley]]

When we implement this, we can remove the hacky workaround added by bug 1059949:
https://github.com/mozilla/treeherder-ui/commit/0bdb50c7f6a49da7ea680ea85fd0ac546f2fae99

Comment 6

[Jonathan Griffin (:jgriffin)]

Clearing the needinfo, this is on our list.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Now that we use the JSON schema for results submitted via Mozilla Pulse, will we also update the client now to use the schema?

Comment 10

[Ed Morley [:emorley]]

Submitting jobs/pushes via the API is now discouraged in favour of using Pulse (bug 1349182), so we're wontfixing API-specific issues.