1224172 - taskcluster-lib-validate: CLI to generate json reference(s) + schemas from node.js source code + yaml for hosted apps

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

In https://github.com/taskcluster/taskcluster-hooks/pull/9 I broke a schema. See:

  * http://schemas.taskcluster.net/hooks/v1/hook-definition.json# (/properties/schedule/description)

This was due to an indentation error in my commit, but I should have validated the generated json before going live. Unfortunately I couldn't work out how to do that; I couldn't work out what commands to run in order to generate the json files locally. I only knew that if I push to heroku, they will get generated and uploaded to production.

This bug therefore has two parts:

1) document the way to validate schema changes before going live (i.e. generate json schemas but don't upload them)
2) fix the indentation mess I made as a result of the PR mentioned above

I think I have a solution for 2) but I depend on 1) which I don't currently have a solution for.

I discovered that the work is performed in https://github.com/taskcluster/schema-validator-publisher but I wasn't sure how to run it in isolation.

I see there is a `publish` boolean which I can set to false, but I'm not sure how to set up the configuration file, as I always get errors locally:



pmoore@Petes-iMac:~/git/taskcluster-hooks master $ export PROFILE=test
pmoore@Petes-iMac:~/git/taskcluster-hooks master $ ./node_modules/babel/bin/babel-node.js bin/main.js validator

/Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js:287
          throw _iteratorError5;
                ^
Unknown assertation failure occured, assumed `undefined` to exist /Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js
                                        v
    272. 
    273.         assume(options[vComp]).exists();
    274.       }
                                        ^
        at Assert.exists (/Users/pmoore/git/taskcluster-hooks/node_modules/assume/index.js:426:15)
        at /Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js:273:32
        at Object.<anonymous> (/Users/pmoore/git/taskcluster-hooks/bin/main.js:106:5)
        at Module._compile (module.js:460:26)
        at normalLoader (/Users/pmoore/git/taskcluster-hooks/node_modules/babel/node_modules/babel-core/lib/api/register/node.js:199:5)
        at Object.require.extensions.(anonymous function) [as .js] (/Users/pmoore/git/taskcluster-hooks/node_modules/babel/node_modules/babel-core/lib/api/register/node.js:216:7)
        at Module.load (module.js:355:32)
        at Function.Module._load (module.js:310:12)
        at Function.Module.runMain (module.js:501:10)

pmoore@Petes-iMac:~/git/taskcluster-hooks master $ 




and ....



pmoore@Petes-iMac:~/git/taskcluster-hooks master $ heroku local
forego     | starting web.1 on port 5000
forego     | starting scheduler.1 on port 5100
scheduler.1 | 
scheduler.1 | Unknown assertation failure occured, assumed `undefined` to exist /Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js
scheduler.1 | le.load (module.js:355:32)
web.1      | 
web.1      | tion) [as .js] (/Users/pmoore/git/taskcluster-hooks/node_modules/babel/node_modules/babel-core/lib/api/register/node.js:216:7)
forego     | sending SIGTERM to web.1
pmoore@Petes-iMac:~/git/taskcluster-hooks master $




I wasn't sure how to overcome these.

I created user-config.yml based on the example file, but that didn't help. I tried also a taskcluster-hooks.conf.json file, but that also didn't get me further. I also logged into heroku and set up all the environment variables I saw there, but that also didn't solve it. I can't quite work out where this error is coming from locally. I wonder if my local node or babel or babel-cli version is incompatible with production. I'm not sure what value it is trying to get from my config, and where I should place that config (env variable, user-config.yml or taskcluster-hooks.conf.json). I also tried reading through the code where the error lies, but it seems to be the transpiled code, which makes my brain bleed when I read it. =)

Rather than spend too much more time on it, I figured best to park it and work on something else, as folks will be around in a few hours that can probably tell me solution immediately. =)

Thanks!

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Pete, what's the actual question here?

Comment 2

[Pete Moore [:pmoore][:pete]]

I wanted to know how to generate the references and schemas json files *without* actually publishing them to production, and I wanted to make sure it is documented and clear for everybody else too.

At the moment we seem only to support an atomic action of <generate json references and schemas and publish them to production>. This is bad as there is no way of testing changes without publishing those changes to production.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

OK, thanks

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

> 1) document the way to validate schema changes before going live
> (i.e. generate json schemas but don't upload them)

Run unit tests that that will load schemas and report any errors by crashing tests..

> I wanted to know how to generate the references and schemas json files *without* actually publishing
> them to production, and I wanted to make sure it is documented and clear for everybody else too.

This happens during "npm test" as a part of the integration tests... as the code loads schemas from disk.
When running tests publish should be false..

---

Problems with getting tests running aren't new... We've started to do user-config-example.yml in many repos... But it's a long way.

---
Let's close this bug.

Comment 5

[Pete Moore [:pmoore][:pete]]

We discussed a little on IRC about the particular use case I was considering:

1) generate the json schemas from the yml templates
2) make a change, e.g. to a constant declaration
3) regenerate the json schemas, and see what has changed

Currently this is not possible, and we agreed it should be, some day. Running all integration tests also takes too much time, and also would not achieve the same.

Comment 6

[Pete Moore [:pmoore][:pete]]

John,

Is this something that would need to be implemented in schema-validator-publisher component, or in (an)other component(s) that use it?

Thanks!
Pete

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

Remember that http://jsonschemalint.com/draft4/ can read YAML for the document and schema.

I think that a tool to do this makes sense.  The code to do the work should go into schema-validator-publisher, and we could have a script entry point in both the schema-validator-publisher package.json and taskcluster-base package.json.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Pete, would you be willing to mentor this?  Otherwise, with nobody working on it, I think we should close it.

Comment 9

[Pete Moore [:pmoore][:pete]]

Yes, I'll mentor it.

Comment 10

[Pavan Karthik [:matrixisreal]]

Hi, I would like to work on this bug.
Could you guide me on getting started?
thanks :D

Comment 11

[Pete Moore [:pmoore][:pete]]

Hi Pavan!

Sure! Let's have an initial video conference to chat about this. Which timezone are you in, and when would be a good time for you? I'm based in Germany, and am certainly available between 9am and 6pm every day, but also can meet up outside of this time, if something else is preferred.

Pete

Comment 12

[Pavan Karthik [:matrixisreal]]

Hi Pete!
Thats sounds great, Im new to TaskCluster and to get familiarized with what it does I might need some Introduction before our discussion. If you think thats fine, we can have a discussion, whenever you say ,Im a college student anyway :P.
Btw I live in India( GMT +5.30)
Thanks :D

Comment 13

[Pete Moore [:pmoore][:pete]]

Hi Pavan!

That's great - then we are only 4.5 hours apart, so my mornings should match your afternoons. Today and tomorrow (Wed/Thu) I'm travelling but how about 14:30 IST / 10:00 CET on Friday 6 Jan?

On skype I am peterjamesmoore - and I am pmoore on irc.mozilla.org (you can find me in #taskcluster channel). If you prefer a different time, let me know ([e-mail]).

No worries about not having knowledge about taskcluster - we can cover that in the meeting.

Thanks!
Pete

Comment 14

[Pavan Karthik [:matrixisreal]]

Sounds Great, See you in Friday then :D.
Ill contact you a day before to remind you or inform any changes (just in case)
Thanks :)

Comment 15

[Pete Moore [:pmoore][:pete]]

Hi Pavan,
I'm on skype - what is your skype name?

Comment 16

[Pavan Karthik [:matrixisreal]]

Hi Pete, 
I am sorry, but due to some technical difficulties with my laptop can.we switch our discussion by 2 or 3 hrs or whatever you say.
Thanks

Comment 17

[Pete Moore [:pmoore][:pete]]

Responded via email...

Comment 18

[Pete Moore [:pmoore][:pete]]

After some

Comment 19

[Pete Moore [:pmoore][:pete]]

After some further investigation, it looks like schema-validator-publisher has been superseded by taskcluster-lib-validate, which splits out publish / render / validate:

https://github.com/taskcluster/taskcluster-lib-validate/tree/master/src

Taskcluster apps which import taskcluster-lib-validate thereby should have access to these atomic functions.

I believe therefore we just need to expose a command to be able to generate a local e.g. build/dist directory which contains the generated references and schemas, and to document this ideally in taskcluster-lib-validate, and in apps that import it, so that anyone hacking on such a taskcluster app can easily discover the process to generate the json artifacts locally, so that when making changes to the node.js source code or schema yaml files, they can see exactly what impact it will have on schemas/references prior to publishing.

Comment 20

[Pete Moore [:pmoore][:pete]]

Brian / John / Jonas / Eli / Dustin,

I'm not sure what a canonical/ideal/idiomatic way to implement this might be in node.js - e.g. if just a script that gets run, or whether to use a framework like grunt to expose an action etc. What do you think?

Thanks!

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

I think it would be a taskcluster-lib-loader endpoint.  For example:
  https://github.com/taskcluster/taskcluster-queue/blob/master/src/main.js#L333

This can be called with

npm run compile && NODE_ENV=.. node lib/main.js scan

Comment 22

[John Ford [:jhford] CET/CEST Berlin Time]

https://github.com/taskcluster/taskcluster-lib-validate/pull/31

This pull request adds an option to the lib-validate library to print out to console or a local file a processed version of the schemas that we'll end up using.

The following environment variables would enable displaying or writing these files.

PREVIEW_JSON_SCHEMA_FILES
WRITE_JSON_SCHEMA_FILES

It shouldn't be too hard to extend this to a command line client, but I suspect that the easier thing is to do a quick start of the required component and take a look at the output in the "rendered_schemas" directory.

Comment 23

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #21)
> I think it would be a taskcluster-lib-loader endpoint.  For example:
>  
> https://github.com/taskcluster/taskcluster-queue/blob/master/src/main.js#L333
> 
> This can be called with
> 
> npm run compile && NODE_ENV=.. node lib/main.js scan

This is functionality that really ought to be built into lib-validate.  Every project really should have access to this without needing to add the same code to their main.js file.

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, I agree.  I'm not sure setting some env vars and running the server is a lot better solution, though!  Is it possible for tc-lib-validate to add a script that would appear in node_modules/.bin?

At any rate, the intent was that Pavan would work on this - has that changed?

Comment 25

[Pavan Karthik [:matrixisreal]]

(In reply to Dustin J. Mitchell [:dustin] from comment #24)
> Yeah, I agree.  I'm not sure setting some env vars and running the server is
> a lot better solution, though!  Is it possible for tc-lib-validate to add a
> script that would appear in node_modules/.bin?
> 
> At any rate, the intent was that Pavan would work on this - has that changed?

Hi Dustin
Apparently from a discussion in IRC, this bug is no more a good first bug, and it requires more knowledge about the existing code.
So, Peete suggested me to work on something else.
But if you still feel, that i Might work on this, I am always ready.
Thanks

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, ok, I didn't realize you and pete had talked, and it looked like you were getting ignored.  Sorry for the confusion!

Comment 27

[Pete Moore [:pmoore][:pete]]

Hi Pavan,

Have you had a chance to browse through any other open issues that might be attractive to work on? Maybe we can chat in #tc-contributors if you get the chance.

This search might uncover some good first bugs:
https://bugzilla.mozilla.org/buglist.cgi?list_id=13391498&status_whiteboard_type=allwordssubstr&status_whiteboard=good&query_format=advanced&product=Taskcluster

Thanks,
Pete

Comment 28

[Pavan Karthik [:matrixisreal]]

Hi Pete,
I guess Ill start working on TaskCluster a bit "later".
I'll let you know when I need some help.
Thanks. :)

Comment 29

[Brian Stack [:bstack]]

Something like this will happen for redeployability I believe! Removing good-first-bug status from this.

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

Pete, how do you see this fitting into a redeployable cluster.  Maybe it's something that tc-references can produce?  Is the lib-validate and env-variables stuff that john landed enough?

Comment 31

[Pete Moore [:pmoore][:pete]]

TBH I'm a bit out of touch with the situation, and how it looks when we are redeployable. What was important for me was that I was able to make changes to references/schemas and regenerate them locally so that I can validate my changes, before pushing them to production.

If that is now possible, we can close this bug.

In a redeployable taskcluster, what is the workflow for making changes to production schemas/references? Is there a means to regenerate all the schemas and references from source (locally), without deploying them?

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

So that is possible now with a single service, thanks to john's work.

If you want to generate all schemas/references, then yes, that is possible -- taskcluster-references does that.

Comment 33

[Pete Moore [:pmoore][:pete]]

\o/ Thanks!