In https://github.com/taskcluster/taskcluster-hooks/pull/9 I broke a schema. See: * http://schemas.taskcluster.net/hooks/v1/hook-definition.json# (/properties/schedule/description) This was due to an indentation error in my commit, but I should have validated the generated json before going live. Unfortunately I couldn't work out how to do that; I couldn't work out what commands to run in order to generate the json files locally. I only knew that if I push to heroku, they will get generated and uploaded to production. This bug therefore has two parts: 1) document the way to validate schema changes before going live (i.e. generate json schemas but don't upload them) 2) fix the indentation mess I made as a result of the PR mentioned above I think I have a solution for 2) but I depend on 1) which I don't currently have a solution for. I discovered that the work is performed in https://github.com/taskcluster/schema-validator-publisher but I wasn't sure how to run it in isolation. I see there is a `publish` boolean which I can set to false, but I'm not sure how to set up the configuration file, as I always get errors locally: pmoore@Petes-iMac:~/git/taskcluster-hooks master $ export PROFILE=test pmoore@Petes-iMac:~/git/taskcluster-hooks master $ ./node_modules/babel/bin/babel-node.js bin/main.js validator /Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js:287 throw _iteratorError5; ^ Unknown assertation failure occured, assumed `undefined` to exist /Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js v 272. 273. assume(options[vComp]).exists(); 274. } ^ at Assert.exists (/Users/pmoore/git/taskcluster-hooks/node_modules/assume/index.js:426:15) at /Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js:273:32 at Object.<anonymous> (/Users/pmoore/git/taskcluster-hooks/bin/main.js:106:5) at Module._compile (module.js:460:26) at normalLoader (/Users/pmoore/git/taskcluster-hooks/node_modules/babel/node_modules/babel-core/lib/api/register/node.js:199:5) at Object.require.extensions.(anonymous function) [as .js] (/Users/pmoore/git/taskcluster-hooks/node_modules/babel/node_modules/babel-core/lib/api/register/node.js:216:7) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10) pmoore@Petes-iMac:~/git/taskcluster-hooks master $ and .... pmoore@Petes-iMac:~/git/taskcluster-hooks master $ heroku local forego | starting web.1 on port 5000 forego | starting scheduler.1 on port 5100 scheduler.1 | scheduler.1 | Unknown assertation failure occured, assumed `undefined` to exist /Users/pmoore/git/taskcluster-hooks/node_modules/taskcluster-lib-loader/lib/loader.js scheduler.1 | le.load (module.js:355:32) web.1 | web.1 | tion) [as .js] (/Users/pmoore/git/taskcluster-hooks/node_modules/babel/node_modules/babel-core/lib/api/register/node.js:216:7) forego | sending SIGTERM to web.1 pmoore@Petes-iMac:~/git/taskcluster-hooks master $ I wasn't sure how to overcome these. I created user-config.yml based on the example file, but that didn't help. I tried also a taskcluster-hooks.conf.json file, but that also didn't get me further. I also logged into heroku and set up all the environment variables I saw there, but that also didn't solve it. I can't quite work out where this error is coming from locally. I wonder if my local node or babel or babel-cli version is incompatible with production. I'm not sure what value it is trying to get from my config, and where I should place that config (env variable, user-config.yml or taskcluster-hooks.conf.json). I also tried reading through the code where the error lies, but it seems to be the transpiled code, which makes my brain bleed when I read it. =) Rather than spend too much more time on it, I figured best to park it and work on something else, as folks will be around in a few hours that can probably tell me solution immediately. =) Thanks!
Pete, what's the actual question here?
I wanted to know how to generate the references and schemas json files *without* actually publishing them to production, and I wanted to make sure it is documented and clear for everybody else too. At the moment we seem only to support an atomic action of <generate json references and schemas and publish them to production>. This is bad as there is no way of testing changes without publishing those changes to production.
Component: Hooks → General
> 1) document the way to validate schema changes before going live > (i.e. generate json schemas but don't upload them) Run unit tests that that will load schemas and report any errors by crashing tests.. > I wanted to know how to generate the references and schemas json files *without* actually publishing > them to production, and I wanted to make sure it is documented and clear for everybody else too. This happens during "npm test" as a part of the integration tests... as the code loads schemas from disk. When running tests publish should be false.. --- Problems with getting tests running aren't new... We've started to do user-config-example.yml in many repos... But it's a long way. --- Let's close this bug.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
We discussed a little on IRC about the particular use case I was considering: 1) generate the json schemas from the yml templates 2) make a change, e.g. to a constant declaration 3) regenerate the json schemas, and see what has changed Currently this is not possible, and we agreed it should be, some day. Running all integration tests also takes too much time, and also would not achieve the same.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Petey bust a schema → CLI to generate json schemas from yaml for taskcluster server components
John, Is this something that would need to be implemented in schema-validator-publisher component, or in (an)other component(s) that use it? Thanks! Pete
Summary: CLI to generate json schemas from yaml for taskcluster server components → schema-validator-publisher: CLI to generate json schemas from yaml for taskcluster server components
Remember that http://jsonschemalint.com/draft4/ can read YAML for the document and schema. I think that a tool to do this makes sense. The code to do the work should go into schema-validator-publisher, and we could have a script entry point in both the schema-validator-publisher package.json and taskcluster-base package.json.
Pete, would you be willing to mentor this? Otherwise, with nobody working on it, I think we should close it.
Yes, I'll mentor it.
Whiteboard: [good first bug]
Hi, I would like to work on this bug. Could you guide me on getting started? thanks :D
Hi Pavan! Sure! Let's have an initial video conference to chat about this. Which timezone are you in, and when would be a good time for you? I'm based in Germany, and am certainly available between 9am and 6pm every day, but also can meet up outside of this time, if something else is preferred. Pete
Hi Pete! Thats sounds great, Im new to TaskCluster and to get familiarized with what it does I might need some Introduction before our discussion. If you think thats fine, we can have a discussion, whenever you say ,Im a college student anyway :P. Btw I live in India( GMT +5.30) Thanks :D
Hi Pavan! That's great - then we are only 4.5 hours apart, so my mornings should match your afternoons. Today and tomorrow (Wed/Thu) I'm travelling but how about 14:30 IST / 10:00 CET on Friday 6 Jan? On skype I am peterjamesmoore - and I am pmoore on irc.mozilla.org (you can find me in #taskcluster channel). If you prefer a different time, let me know (email@example.com). No worries about not having knowledge about taskcluster - we can cover that in the meeting. Thanks! Pete
Sounds Great, See you in Friday then :D. Ill contact you a day before to remind you or inform any changes (just in case) Thanks :)
Hi Pavan, I'm on skype - what is your skype name?
Hi Pete, I am sorry, but due to some technical difficulties with my laptop can.we switch our discussion by 2 or 3 hrs or whatever you say. Thanks
Responded via email...
Summary: schema-validator-publisher: CLI to generate json schemas from yaml for taskcluster server components → taskcluster-lib-validate: CLI to generate json schemas from yaml for taskcluster server components
After some further investigation, it looks like schema-validator-publisher has been superseded by taskcluster-lib-validate, which splits out publish / render / validate: https://github.com/taskcluster/taskcluster-lib-validate/tree/master/src Taskcluster apps which import taskcluster-lib-validate thereby should have access to these atomic functions. I believe therefore we just need to expose a command to be able to generate a local e.g. build/dist directory which contains the generated references and schemas, and to document this ideally in taskcluster-lib-validate, and in apps that import it, so that anyone hacking on such a taskcluster app can easily discover the process to generate the json artifacts locally, so that when making changes to the node.js source code or schema yaml files, they can see exactly what impact it will have on schemas/references prior to publishing.
Brian / John / Jonas / Eli / Dustin, I'm not sure what a canonical/ideal/idiomatic way to implement this might be in node.js - e.g. if just a script that gets run, or whether to use a framework like grunt to expose an action etc. What do you think? Thanks!
Summary: taskcluster-lib-validate: CLI to generate json schemas from yaml for taskcluster server components → taskcluster-lib-validate: CLI to generate json reference(s) + schemas from node.js source code + yaml for hosted apps
I think it would be a taskcluster-lib-loader endpoint. For example: https://github.com/taskcluster/taskcluster-queue/blob/master/src/main.js#L333 This can be called with npm run compile && NODE_ENV=.. node lib/main.js scan
https://github.com/taskcluster/taskcluster-lib-validate/pull/31 This pull request adds an option to the lib-validate library to print out to console or a local file a processed version of the schemas that we'll end up using. The following environment variables would enable displaying or writing these files. PREVIEW_JSON_SCHEMA_FILES WRITE_JSON_SCHEMA_FILES It shouldn't be too hard to extend this to a command line client, but I suspect that the easier thing is to do a quick start of the required component and take a look at the output in the "rendered_schemas" directory.
(In reply to Dustin J. Mitchell [:dustin] from comment #21) > I think it would be a taskcluster-lib-loader endpoint. For example: > > https://github.com/taskcluster/taskcluster-queue/blob/master/src/main.js#L333 > > This can be called with > > npm run compile && NODE_ENV=.. node lib/main.js scan This is functionality that really ought to be built into lib-validate. Every project really should have access to this without needing to add the same code to their main.js file.
Yeah, I agree. I'm not sure setting some env vars and running the server is a lot better solution, though! Is it possible for tc-lib-validate to add a script that would appear in node_modules/.bin? At any rate, the intent was that Pavan would work on this - has that changed?
(In reply to Dustin J. Mitchell [:dustin] from comment #24) > Yeah, I agree. I'm not sure setting some env vars and running the server is > a lot better solution, though! Is it possible for tc-lib-validate to add a > script that would appear in node_modules/.bin? > > At any rate, the intent was that Pavan would work on this - has that changed? Hi Dustin Apparently from a discussion in IRC, this bug is no more a good first bug, and it requires more knowledge about the existing code. So, Peete suggested me to work on something else. But if you still feel, that i Might work on this, I am always ready. Thanks
Ah, ok, I didn't realize you and pete had talked, and it looked like you were getting ignored. Sorry for the confusion!
Hi Pavan, Have you had a chance to browse through any other open issues that might be attractive to work on? Maybe we can chat in #tc-contributors if you get the chance. This search might uncover some good first bugs: https://bugzilla.mozilla.org/buglist.cgi?list_id=13391498&status_whiteboard_type=allwordssubstr&status_whiteboard=good&query_format=advanced&product=Taskcluster Thanks, Pete
Flags: needinfo?(pmoore) → needinfo?(pavankarthikboddeda)
Hi Pete, I guess Ill start working on TaskCluster a bit "later". I'll let you know when I need some help. Thanks. :)
Something like this will happen for redeployability I believe! Removing good-first-bug status from this.
Whiteboard: [good first bug]
You need to log in before you can comment on or make changes to this bug.