1156319 - Define deployment orchestration of taskcluster stack

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Pete Moore [:pmoore][:pete]]

Using either docker orchestration tools, aws cloud formation, or some similar technology, define an orchestrated deployment of as much of the task cluster stack as possible, to simplify and standardise taskcluster setup.

The measurable outcomes of this bug would be:

* Is it trivial to deploy a task cluster stack locally for development?
* Is it easy to spin up a new task cluster environment in the cloud?

Pulse/AMQP setup is likely to be non trivial, and might not be an initial deliverable. However, a simple deployment of Queue, Scheduler (so long as it exists), Provisioner, Docs, Status, Tools, the reference schemas and manifests, would be a useful start.

Comment 1

[Pete Moore [:pmoore][:pete]]

Q2 has now come and gone, so no need to track this as a Q2 stretch goal any more.

Comment 2

[Pete Moore [:pmoore][:pete]]

The plan is to start attacking this in 2018 Q1. \o/

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

At the AllHands we talked about this being split into multiple stages
illustrated as follows:

>  [source] -> build -> [binaries] -> deploy -> [cluster]
>                ^                       ^          ^--------- (runtime configuration/data lives here)
>               (build configuration)   (deployment configuration)
 , where:

build configuration:
   * services to enable / disable
   * docker images hashes pinning images to be pulled
   * pinning of AMIs
   * default configuration values for some deployment config options
   * ...
deployment configuration:
   * aws root credentials
   * ec2 regions to use (things like that)
   * domain name
   * built-in runtime configuration, such as: roles, clients, secrets, workerTypes,
     (built-in runtime configuration cannot be modified at runtime)
runtime configuration:
   (these are things that can be changed at runtime)
   * roles, clients, workerTypes, hooks, etc.
runtime data:
   (ideally, this is immutable)
   * tasks, artifacts, index entries

[source]: (the input when building)
   * A github repository referencing all our repositories
   * Scripts for building docker images, AMIs using our repositories
   * Scripts for building a Helm recipe for deployment on kubernetes
   * Scripts for building a terraform module for deploying taskcluster using kubernetes

[binaries]: (the output from building, input for deployment)
   * docker images, AMIs, and other machine images
   * Helm recipe for deployment on kubernetes
   * terraform module for deployment an entire cluster

[cluster]: (the result of deployment)
   * A collection of taskcluster services and workers

Notice:
  terraform is needed for cross-cloud provisioning, setup of not just core
  taskcluster services, but also:
    * built-in worker-types,
    * cross-region replication of object storage,
    * webhooktunnel (maybe),
    * other snowflakes that don't fit in k8s

  The Helm recipe will probably give a set of taskcluster core services, without
  advanced things like EC2 spot workers, etc. But this will be sufficient for
  local development of the services.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Brian, is this something you're working on?

Comment 5

[Brian Stack [:bstack]]

https://github.com/taskcluster/taskcluster-terraform && https://github.com/taskcluster/taskcluster-mozilla-terraform

Both exist and "work" still need to make them work better, but this no longer blocks things.