1522477 - monorepo: yarn run generate

Status: RESOLVED FIXED

(Taskcluster :: Services, enhancement)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This is still pretty vague, but I'd like a place to toss ideas as they come.

We should do some code generation in the monorepo, in such a way that the generated code is checked in. Another way to look at that is, rather than try to write everything exactly once in the repo, let's automate changes to the multiple places that we need to write things.

The main things I want to generate are:

• references (manifest, schemas, and api/exchange references)
• docs (stuff that ends up in "References" in the docs, so README, docs/*)
• client(s) (so that service A can call a new method in service B after yarn run generate)

I'd like to build an extensible generation system, so that we can add more generated things later. These could even be fairly minor things like ensuring that node versions match everywhere we need to write them.

In CI, we'll run yarn run generate and, if any git diffs appear, fail the task -- thereby assuring that the generated code is always correct when landed.

Docs probably won't be generated forever -- there's no need to have two copies of every markdown file in the same repo -- but until tc-docs and tc-web are unified this will make things easier for the build process.

I'd like to generate the client(s) in such a way that they can be published from the monorepo. That should at least include the JS and Web clients, but could potentially include others -- discussion is underway.

While I'm at it, I'd like to move the existing taskcluster-builder command-line tool (which only has a single subcommand) to be invoked with yarn build.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 1516351 also mentions that this will mean diffs can be analyzed. So we could have something run on PRs that fails if schemas change incompatibly (that is, if there exists a document which was valid with the old schema but not valid with the new). That might be a fun project for some comp-sci student :)

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

https://github.com/taskcluster/taskcluster/pull/165

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

https://tools.taskcluster.net/groups/MaZOhUlMRAy5WZiuxRbSZg/tasks/BTPpswcJRxaqOdfIhAwG5g/runs/0/logs/public%2Flogs%2Flive_backing.log shows git diff displaying via more and timing out. Whoops. Also, I think the bunch of PRs that landed at about the same time yesterday caused generation to fail.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

https://github.com/taskcluster/taskcluster/pull/186