1092279 - dockerhost: Create new worker type which exposes a docker host (mostly) securely

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[James Lal [:lightsofapollo]]

There are tons of potential uses but the current case is builds are slow locally in emulated environments and we are doing tons of network bound work in addition which will never be fast locally. An easy way to work around this is simply using remote docker instances. This means we can reuse the existing docker tooling (even running tests on these instances!).

Here is my plan:

 - The worker will create its own keys (https://docs.docker.com/articles/https/) and expose a port which will require tls verification (which requires a specific client key) this port will be open to the world (but nobody can access it yet)

 - The worker will listen over amqp for a task and create an artifact which will point to the authentication url / api.json. Each worker will _only_ work one single task (see below).

 - There will be a tiny server side component which validates a taskcluster client based request (and that is all it can do) and return copies of the keys needed to access the worker

 - A CLI based tool will abstract much of this away as long as you have a valid TC client. It will end up generating a script called something like "docker-tc" which will wrap your local docker client in the correct keys.

 - "docker-tc" will function identically do your local docker client (volume mounts will work but will happen on the remote docker host!)

 - Only _one_ certificate will be issued ever meaning only one user can access the docker-host (meaning each worker can only ever consume _one_ task)

Work started here: https://github.com/lightsofapollo/dockerhost-worker

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

> There will be a tiny server side component which validates a taskcluster client based request
> (and that is all it can do) and return copies of the keys needed to access the worker
I assume the client certificate consists of a public and private key, why don't we just put the public key
into the task.payload section?
That way the client just does as follows:
 1. Generate public/private key
 2. Create task with public key in payload
 3. Listen for artifact-created with given taskId
 4. Get host:port for docker-host from artifact
 5. Connect to host:port using private key

Note, If you want to make this pretty you can dearmor the public key before stuffing it into task.payload.

Comment 2

[James Lal [:lightsofapollo]]

Its a bit more complicated then that https://docs.docker.com/articles/https/

Also the primary reason here is the keys generated on OSX misbehave (according to their docs) so it seems easier to do it server side (this turned out to be pretty easy).

Comment 3

[James Lal [:lightsofapollo]]

Initial version landed instructions here: https://github.com/lightsofapollo/dockerhost-client

More work is needed to make this "production ready"

Comment 4

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...