break taskcluster-base into component libraries and create a taskcluster-platform package that depends on a known-to-work set of component libraries

RESOLVED FIXED

Status

Taskcluster
Platform Libraries
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jhford, Unassigned)

Tracking

Details

Attachments

(3 attachments, 3 obsolete attachments)

taskcluster-base has many components.  We want to split things out into smaller components to make it easier to reuse the libraries externally and also to allow people to selectively upgrade one component without upgrading all, or using an old library instead of the latest version of all components.  We can also stop including LegacyEntity and the future LegacyConfig in all components.

All of the repo splitting for taskcluster-base should be done using git filter-branch to keep history of each component.  We should use babel to statically transpile the code for all the components so that we don't need to use babel-node any longer.  Libraries should use pre-publish.  We should put all the code in a src/ and the output into lib/

Here's an idea of how things would look in the future.  Naming of component libraries is not set in stone yet.  

taskcluster-base will depend on:
    influxing             (rename of base.stats)
    azure-entities        (rename of base.Entity)
    slugid
    yaml-configurator     (rename of base.config)
    declarative-api       (rename of base.API)
    pulse-publisher       (rename of base.Exchanges)
    schema-loader         (rename of base.validator)
    taskcluster-utils     (rename of base.utils)

taskcluster-legacy-entity will be removed from taskcluster-base


In https://github.com/taskcluster/taskcluster-base/blob/master/index.js, instead of iterating of a list of things that are part of taskcluster-base itself, we'd be require()ing the component libraries specified in package.json and exporting those.
Created attachment 8671847 [details]
split.js
Created attachment 8671848 [details]
config.json
Hey, so I have a quick script that does a lot of the mechanical parts of this conversion.  Having this in a script is great because it means that we can verify that everything works well.

The main things which need to happen still are:
  1. pair down the dependencies and dev-dependencies based on the name of the dep occuring in a file.  Maybe we can do something where require('x') in a non-test/ file ensures inclusion in final package.json and require('x') in test/ but only in test/ ensures it's a devDependency and print out a list of dependencies/devDependencies names which occur as a string in their respective directories

  2. Unit testing... We should automatically change require('../') in test files to require('taskcluster-base') and add taskcluster-base as a devDependency

  3. Versioning.  We can either given 0.8.6 to the split out components or we could do something crazy and give all the split out things 1.0.0 and leave tc-base at 0.8.6

  4. tc-base cleanup:  we need to change index.js to require the taskcluster-lib-* instead of the inside files.  We can also |git rm| the files we put in the split repos until we're totally split and then tc-base will become a tiny shell of a repo.  When that happens, we can delete the history and start it anew as just a package.json and index.js repo

I think 1. and 2. can at least partially be automated  

What do you think?
Flags: needinfo?(jopsen)
Created attachment 8671879 [details]
clean-deps.js
with clean-deps.js and split.js, I can automagically convert stats into its own library.

Process:

cd splitting
npm install promise lodash file
git clone https://github.com/taskcluster/taskcluster-base
wget split.js
wget config.json
wget clean-deps.js
babel-node split.js
cd taskcluster-lib-stats
babel-node ../clean-deps.js
npm install .
npm test

This is sort of rough on the edges and not such a great design, but it works where it counts and we only need it to split out a couple of things.
Created attachment 8671882 [details]
clean-deps.js
Attachment #8671879 - Attachment is obsolete: true
Created attachment 8671897 [details]
split.js
Attachment #8671847 - Attachment is obsolete: true
Created attachment 8671898 [details]
config.json
Attachment #8671848 - Attachment is obsolete: true
Also move over taskcluster-lib-config.  Not that because we used to require('../') in the tests, we're doing some pretty silly things here, like testing the config in the taskcluster-base dependency.  We'll need to have someone verify the output of cleanup-deps and split to make sure they make sense in context, but this is 95% of the work of the splitting out.
Oh, and note to self: make sure that for packages that have babel-runtime, that we install a version of babel-runtime!
sorry for the noise: https://github.com/jhford/taskcluster-base-split
> 2. Unit testing... We should automatically change require('../') in test files to
> require('taskcluster-base') and add taskcluster-base as a devDependency
Well, yes when we're testing with another component from tc-base (or dev-depend on the module we need).
Just watch out, stats tests are written to test base.stats, so you can't do:
  base = require('tascluster-base');
As you'll be testing against an old version of stats :)

> 3. Versioning.  We can either given 0.8.6 to the split out components or we could do something 
> crazy and give all the split out things 1.0.0 and leave tc-base at 0.8.6
Go crazy and do what you would like. base.API, base.config base.Exchanges are not quiet stable yet,
I have breaking changes in the pipe for the those (but feel to do what you like).

> What do you think?
I think you're on fire, keep up the good work going!

----
By the way, I like the names you came up with...
I assume we'll taskcluster-lib- prefix the TC specific ones.
Flags: needinfo?(jopsen)
I think we're done with this... All credits to jhford!
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.