1151886 - Alternative json validation engine for TaskCluster services?

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

Today bhearsum had a problem troubleshooting a json validation error, and so I got thinking about ways we might make this easier:

http://logs.glob.uno/?c=mozilla%23taskcluster#c899

Basically he was getting a response like this, that was not *so* easy to read, although might technically be providing sufficient information to explain the validation error:

Received HTTP Payload: {
  "message": "Request payload must follow the schema: http://schemas.taskcluster.net/scheduler/v1/task-graph.json#",
  "error": {
    "validation": {
      "tasks": {
        "schema": {
          "0": {
            "schema": {
              "taskId": {
                "pattern": true
              },
              "task": {
                "schema": {
                  "provisionerId": {
                    "required": true
                  },
                  "taskGroupId": {
                    "pattern": true
                  },
                  "provisionId": {
                    "additional": true
                  }
                }
              }
            }
          },
          "1": {
            "schema": {
              "taskId": {
                "pattern": true
              },
              "task": {
                "schema": {
                  "provisionerId": {
                    "required": true
                  },
                  "taskGroupId": {
                    "patter (limit 1024 char)


The rest of the response body was not captured - this was the beginnings only.

When running a validation check using:
https://json-schema-validator.herokuapp.com/

The explanation of the validation failure seemed more succinct, and thus easier to understand:

[ {
  "level" : "error",
  "schema" : {
    "loadingURI" : "http://schemas.taskcluster.net/queue/v1/create-task-request.json#",
    "pointer" : ""
  },
  "instance" : {
    "pointer" : "/tasks/0/task"
  },
  "domain" : "validation",
  "keyword" : "additionalProperties",
  "message" : "object instance has properties which are not allowed by the schema: [\"provisionId\"]",
  "unwanted" : [ "provisionId" ]
}, {
  "level" : "error",
  "schema" : {
    "loadingURI" : "http://schemas.taskcluster.net/queue/v1/create-task-request.json#",
    "pointer" : ""
  },
  "instance" : {
    "pointer" : "/tasks/0/task"
  },
  "domain" : "validation",
  "keyword" : "required",
  "message" : "object has missing required properties ([\"provisionerId\"])",
  "required" : [ "created", "deadline", "metadata", "payload", "provisionerId", "workerType" ],
  "missing" : [ "provisionerId" ]
}, {
  "level" : "error",
  "schema" : {
    "loadingURI" : "#",
    "pointer" : "/properties/tasks/items/properties/taskId"
  },
  "instance" : {
    "pointer" : "/tasks/0/taskId"
  },
  "domain" : "validation",
  "keyword" : "pattern",
  "message" : "ECMA 262 regex \"^[a-zA-Z0-9-_]{22}\" does not match input string \"l10n_tag\"",
  "regex" : "^[a-zA-Z0-9-_]{22}",
  "string" : "l10n_tag"
} ]


The engine behind this online service is https://github.com/fge/json-schema-validator - unfortunately a java application rather than a javascript solution, so maybe not ideal for integration. However, maybe a javascript port exists, or we could consider other json schema validation engines that might produce similar type output.

This bug is rather subjective, I'm curious if others also favour the output of https://github.com/fge/json-schema-validator or not.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

Agree... I've been looking at is-my-json-valid which compiles JSON schemas into a javascript function :)
See: https://github.com/mafintosh/is-my-json-valid

It's supposedly extremely fast. Which is important to me considering how many things we validate.
We validate every single output and input. So speed is a factor we need to consider.
(Granted I fear we're bounded by the stupidity of our I/O).

So far is-my-json-valid have been missing a feature to substitute in "default" values from the schemas.
I added this today, in PR: https://github.com/mafintosh/is-my-json-valid/pull/55
Along with a few other PRs improving other less relevant issues.

is-my-json-valid might have some issues with $ref, but if that is the case I think we should fix them and submit PRs. The project is fairly small ~500 lines of code.

@pmoore, if you're interested please feel free to sanity check my PRs... And evaluate if is-my-json-valid has missing features and/or how we can fix them. I haven't really tried it's $ref feature.
Note: we never load $ref dynamically, we always have a dictionary with {<schema-url>: <schema>} when we load our schemas. I.e. if a schema is remote we'll download it before we validate anything, there is no reason a thing like is-my-json-valid should fetch resources over HTTP(S).

I'm not above forking if we can't land our PRs, it's a small project. As long as we keep our schemas close the spec we can also move the another engine if we find too many bugs and realize that maintaining a fork is too much work.

@pmoore, Also take a look at their error codes. Again, it's a simple project we can probably improve them. I like that they do a json-path like thingy + message + value in "verbose" mode.
Example error: {field: 'data.hello', message: 'is the wrong type', value: 100}

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

bstack did this! YAY

--
This bug is closed as part of the Great Bugzilla Cleaning Sprint of 2016 by bstack and jonasfj, please reopen if you disagree.