1800016 - Support indexing taskcluster taskgraph yaml files

Status: NEW

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

Although taskcluster does have a bunch of JSON schemas, there are currently none for taskgraph. I asked in https://chat.mozilla.org/#/room/#taskcluster:mozilla.org and https://github.com/taskcluster/taskgraph/issues/148 was filed as a result, so we can check on that before implementing anything, but in general the idea is:

• JSON schemas are a big deal and are extensible, including via meta-schemas.
  • The schemas are conceptually more syntactic in nature, defining the type of something and validation rules which can largely be thought of as expressing constraints on the contents/formatting of a string or requiring/forbidding fields based on the presence of other fields.
• They do not as much specify semantic things like "this value exists in conceptual namespace FOO" other than the string pattern/formatting constraints.
  • But we could probably overlay such a thing ourselves via a meta-schema.
  • Or we could have the semantics defined outside of the JSON schema but effectively using a JSON Pointer syntax/similar to reference the specific node/production in the JSON schema that we want to impose the semantics on.
• JSON schemas are also the basis for YAML validation as used by https://github.com/redhat-developer/vscode-yaml and similar mechanisms, which is relevant for the taskcluster taskgraph representation.
• In general it seems like it would be useful for searchfox's understanding to be able to help explain what values are too beyond just cross-referencing, plus JSON schemas could then also work on JSON files, etc.
  • Also there are things like Web Extensions that are defined in terms of JSON Schemas; bug 1748714 is an example thread that could be followed.

The tentative plan for parsing would be to use tree-sitter-yaml whose playground at https://ikatyang.github.io/tree-sitter-yaml/ worked when I last tested it but the more general tree-sitter playground at https://tree-sitter.github.io/tree-sitter/playground did not work for YAML. There doesn't seem to be a lot of parsers that are otherwise interested in providing the AST level information we'd be interested in, and we probably are already looking at using switching to tree-sitter for tokenization (bug 1796870).