taskcluster-base has many components. We want to split things out into smaller components to make it easier to reuse the libraries externally and also to allow people to selectively upgrade one component without upgrading all, or using an old library instead of the latest version of all components. We can also stop including LegacyEntity and the future LegacyConfig in all components. All of the repo splitting for taskcluster-base should be done using git filter-branch to keep history of each component. We should use babel to statically transpile the code for all the components so that we don't need to use babel-node any longer. Libraries should use pre-publish. We should put all the code in a src/ and the output into lib/ Here's an idea of how things would look in the future. Naming of component libraries is not set in stone yet. taskcluster-base will depend on: influxing (rename of base.stats) azure-entities (rename of base.Entity) slugid yaml-configurator (rename of base.config) declarative-api (rename of base.API) pulse-publisher (rename of base.Exchanges) schema-loader (rename of base.validator) taskcluster-utils (rename of base.utils) taskcluster-legacy-entity will be removed from taskcluster-base In https://github.com/taskcluster/taskcluster-base/blob/master/index.js, instead of iterating of a list of things that are part of taskcluster-base itself, we'd be require()ing the component libraries specified in package.json and exporting those.
Hey, so I have a quick script that does a lot of the mechanical parts of this conversion. Having this in a script is great because it means that we can verify that everything works well. The main things which need to happen still are: 1. pair down the dependencies and dev-dependencies based on the name of the dep occuring in a file. Maybe we can do something where require('x') in a non-test/ file ensures inclusion in final package.json and require('x') in test/ but only in test/ ensures it's a devDependency and print out a list of dependencies/devDependencies names which occur as a string in their respective directories 2. Unit testing... We should automatically change require('../') in test files to require('taskcluster-base') and add taskcluster-base as a devDependency 3. Versioning. We can either given 0.8.6 to the split out components or we could do something crazy and give all the split out things 1.0.0 and leave tc-base at 0.8.6 4. tc-base cleanup: we need to change index.js to require the taskcluster-lib-* instead of the inside files. We can also |git rm| the files we put in the split repos until we're totally split and then tc-base will become a tiny shell of a repo. When that happens, we can delete the history and start it anew as just a package.json and index.js repo I think 1. and 2. can at least partially be automated What do you think?
with clean-deps.js and split.js, I can automagically convert stats into its own library. Process: cd splitting npm install promise lodash file git clone https://github.com/taskcluster/taskcluster-base wget split.js wget config.json wget clean-deps.js babel-node split.js cd taskcluster-lib-stats babel-node ../clean-deps.js npm install . npm test This is sort of rough on the edges and not such a great design, but it works where it counts and we only need it to split out a couple of things.
Created attachment 8671882 [details] clean-deps.js
Also move over taskcluster-lib-config. Not that because we used to require('../') in the tests, we're doing some pretty silly things here, like testing the config in the taskcluster-base dependency. We'll need to have someone verify the output of cleanup-deps and split to make sure they make sense in context, but this is 95% of the work of the splitting out.
Oh, and note to self: make sure that for packages that have babel-runtime, that we install a version of babel-runtime!
sorry for the noise: https://github.com/jhford/taskcluster-base-split
> 2. Unit testing... We should automatically change require('../') in test files to > require('taskcluster-base') and add taskcluster-base as a devDependency Well, yes when we're testing with another component from tc-base (or dev-depend on the module we need). Just watch out, stats tests are written to test base.stats, so you can't do: base = require('tascluster-base'); As you'll be testing against an old version of stats :) > 3. Versioning. We can either given 0.8.6 to the split out components or we could do something > crazy and give all the split out things 1.0.0 and leave tc-base at 0.8.6 Go crazy and do what you would like. base.API, base.config base.Exchanges are not quiet stable yet, I have breaking changes in the pipe for the those (but feel to do what you like). > What do you think? I think you're on fire, keep up the good work going! ---- By the way, I like the names you came up with... I assume we'll taskcluster-lib- prefix the TC specific ones.
I think we're done with this... All credits to jhford!