893080 - Automate the setup of new development masters and associated slaves in AWS (pods)

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

The current dev-master VM has never served us particularly well for development: everyone from releng has _at least_ one master setup and resource contention is often an issue.

At the past few work weeks, we discussed the concept of a developer "pod": a collection of VMs that provide all the necessary services required to do development work on buildbot and/or our configs. All the various VMs in a given pod (buildbot master, db, linux slaves, staging server?, win64 slaves?, redis?) would be stood up in AWS in an automated way, and other platforms (read: hardware slaves) could then be connected by hand when required.

The automation would need to be able to update the slavealloc entry for the user's master to point to the new location, since this would likely change between invocations.

There should also be an automated way to disable and delete and entire pod, returning any hardware slaves to their pool of origin in so doing.

It's conceivable we could also then use the same automation to loan a pod to a developer if the developer required access to the end-to-end infra was working on an integration issue.