1157769 - It should be trivial to see packer config for how an ami was created

Status: REOPENED

(Taskcluster :: Workers, defect)

Description

[Pete Moore [:pmoore][:pete]]

We currently lack transparency of how an ami was created, that is used by a given workerType.

It would be useful to:
a) have an overview of available amis
b) for each ami, to be able to see the packer config used (or the git commit of docker-worker that the ami was created from)

The goal is that someone with no inside knowledge can ascertain exactly what an ami contains, and how it differs to other available amis.

It might also be useful to show worker types that are associated to a given ami (at the moment it is not easy to have an overview if we just have a couple of amis that are used across all the worker types, or if each worker type typically is using a different ami).

Another option might be that the worker type specifies e.g. a packer config, and something else takes care of generating amis when a new packer config is used that has no existing ami.

Separately we could maintain a mapping of packer configs to amis, and this would make it much more transparent to understand how worker types are configured, and also make it easier to create new ones without needing to worry about the publishing-a-new-ami steps.

Comment 1

[Chris Cooper [:coop] (he/him)]

Found in triage.

Pete: did Wander's recent work on docker-worker deployments fix this for you?

Comment 2

[Pete Moore [:pmoore][:pete]]

I don't think so - I think we should revisit this topic as part of redeployability, when we get to workers.

I think the argument still holds.

I'll see if I can turn it into an RFC.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

This would probably now apply to how generic-worker is built and deployed.  Pete, you already have a needinfo here so I can't add it, but .. does this bug still make sense?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I've been thinking that it might be nice in the worker-manager to allow looking up AMIs by tags.  For GCP, we have image names, which we can make reasonably descriptive (gw-v11-1-0-debian-9-985ab89ef for example at least says what version of livelog was installed).  As part of the build process we could add other metadata such as the livelog version to the image as tags (or in GCP, in the description).

However, the build process is still a giant ❓ in general (I'm only working on the "getting started" workers in bug 1502371), so I'm not sure how this would work..

Comment 5

[Pete Moore [:pmoore][:pete]]

This is solved by the generic-worker to docker-worker migration, since generic-worker tasks link to a source file which describes how the host machine was set up.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Pete, there might be some good ideas here to mix into discussions of how we recommend users handle image generation..

Comment 7

[Pete Moore [:pmoore][:pete]]

We could make the workers (generic-worker/docker-worker) have a required config property, which is a link to the host environment definition, which gets reported in the task log header of all task runs they execute.

Currently the capability to provide the link to the host environment exists in generic-worker, but it isn't enforced.

I don't think such a feature exists in docker-worker.

I have mixed feelings about whether it should be enforced or not by the worker, but I strongly feel for worker types that we manage on the community cluster, we should provide this information, and that docker-worker should also support the feature. I think it is reasonable to expect this of the firefox cluster too.

Comment 8

[Pete Moore [:pmoore][:pete]]

Not actively working on this at the moment.

Comment 9

[Pete Moore [:pmoore][:pete]]

Reopening inactive bugs, because they may still need attention. Historically, inactive bugs were closed, but this hides the fact there are genuine issues which have not been resolved.