1580570 - [image-generation] automated baking of worker images

Status: NEW

(Taskcluster :: Workers, enhancement)

Description

[Miles Crabill [:miles]]

This bug covers automating the generation of production taskcluster worker images.

At a high level, we will need:

• Workers in AWS and GCP with cloud permissions to spin up other instances, snapshot, etc.
• Due to security concerns, these instances should probably live in an independent AWS account from taskcluster-prod, and a separate GCP project
• Support retrying image generation failures

This probably looks like a main task that farms out individual builder tasks to generate images, so that each can be retried individually.

Comment 1

[Brian Stack [:bstack]]

Attachment: Bug 1580570 - Add a ci workerpool for monopacker

Comment 2

[Miles Crabill [:miles]]

We have a worker-pool created for this, but a few more steps to take:

• we need AWS and GCP credentials for baking images that workers in the worker-pool can access (stored as tc secrets)
  • it would probably be easiest to use the taskcluster-staging AWS account for this as it doesn't require creating a new account
    • see bug 1580571
• implement the task that builds images, or a decision style task that triggers image building subtasks
• more, probably

Comment 3

[Brian Stack [:bstack]]

Attachment: Bug 1580570 - Update to existing machine type for tc image building

Comment 4

[Brian Stack [:bstack]]

Attachment: Bug 1580570 - Turn off autorestart for gcp instances

Comment 5

[Miles Crabill [:miles]]

We have a preliminary pass of this baking GCP images in taskcluster here: https://github.com/taskcluster/monopacker/pull/22.

Comment 6

[Miles Crabill [:miles]]

Here is an example task triggered by a commit to the master branch of monopacker:

• https://tools.taskcluster.net/groups/NAttq-lFQsCe2G1O4Cr8kg/tasks/CL1m6y5gT86JhHZnd5xurw/details

The task has an artifact generated by Packer that details the output image IDs. For now, do not have production secrets in place - that is the next and hopefully last step towards real production images being generated automatically in CI.

Comment 7

[Miles Crabill [:miles]]

We've run into a blocker for automating this process in taskcluster: storing the CoT key in taskcluster secrets violates the independent second factor intent of chain of trust.

Per aki: "The CoT key is supposed to be a second factor, separate from Taskcluster scopes. If it's deployable with Taskcluster scopes, it's no longer an independent second factor."

There are changes to the chain of trust implementation in the works that would make automating this in the taskcluster platform more feasible in the future.

For now, any kind of automation can be used to trigger monopacker - as long as there is a secure place to store secrets and run jobs that is independent of taskcluster, that should be sufficient.