Bugzilla

Currently:

OCC required additional action to deploy {gecko,comm}-{1,2,3}/b-win2012 images with the new fingerprint. Levels 1 and 2 (untrusted) is now rebuilt and are going green. Level 3 requires a CoT key. :markco doesn't have this key and doesn't know how to create a new image with the trusted CoT key. This means that our trusted level 3 builders are stuck on the old hgmo fingerprint, which resulted in the Firefox 82.0 RC graph being stuck.

I saw three potential solutions:

1. Roll back the fingerprint change. Because the new images have both the new and previous fingerprints, they should still work. Because the previous image has the previous fingerprint, those should start working again. However, the fingerprint will expire tomorrow. This is not a good solution after midnight UTC tonight.
2. Find the current CoT private key, figure out how to add it to the image (populate the taskcluster secret? what format should this be in?), and rerun the level 3 image builder task. This may need to wait for :grenade.
3. Generate a new CoT keypair, add it to `scriptworker.constants`, roll out a new scriptworker release + k8s images, then populate the new builder images with the new private key.

It looks like we're doing a combination of (1) and (2). :dhouse rolled back the fingerprint change to unblock Firefox 82.0 RC. And :markco will check with :grenade on how to roll out the images with the existing CoT key. (3) can be a fallback option if we no longer have the existing CoT key. We likely won't be able to fully resolve this issue until tomorrow (October 13), but we have some workarounds.

Currently:

OCC required additional action to deploy {gecko,comm}-{1,2,3}/b-win2012 images with the new fingerprint. Levels 1 and 2 (untrusted) is now rebuilt and are going green. Level 3 requires a CoT key. :markco doesn't have this key and doesn't know how to create a new image with the trusted CoT key. This means that our trusted level 3 builders are stuck on the old hg.m.o fingerprint, which resulted in the Firefox 82.0 RC graph being stuck.

I saw three potential solutions:

1. Roll back the fingerprint change. Because the new images have both the new and previous fingerprints, they should still work. Because the previous image has the previous fingerprint, those should start working again. However, the fingerprint will expire tomorrow. This is not a good solution after midnight UTC tonight.
2. Find the current CoT private key, figure out how to add it to the image (populate the taskcluster secret? what format should this be in?), and rerun the level 3 image builder task. This may need to wait for :grenade.
3. Generate a new CoT keypair, add it to `scriptworker.constants`, roll out a new scriptworker release + k8s images, then populate the new builder images with the new private key.

It looks like we're doing a combination of (1) and (2). :dhouse rolled back the fingerprint change to unblock Firefox 82.0 RC. And :markco will check with :grenade on how to roll out the images with the existing CoT key. (3) can be a fallback option if we no longer have the existing CoT key. We likely won't be able to fully resolve this issue until tomorrow (October 13), but we have some workarounds.

After this is done, we should update the documentation about how to roll out a new hg.m.o fingerprint with these additional steps.

Currently:

OCC required additional action to deploy {gecko,comm}-{1,2,3}/b-win2012 images with the new fingerprint. Levels 1 and 2 (untrusted) is now rebuilt and are going green. Level 3 requires a CoT key. :markco doesn't have this key and doesn't know how to create a new image with the trusted CoT key. This means that our trusted level 3 builders are stuck on the old hg.m.o fingerprint, which resulted in the Firefox 82.0 RC graph being stuck.

I saw three potential solutions:

1. Roll back the fingerprint change. Because the new images have both the new and previous fingerprints, they should still work. Because the previous image has the previous fingerprint, those should start working again. However, the fingerprint will expire tomorrow. This is not a good solution after midnight UTC tonight.
2. Find the current CoT private key, figure out how to add it to the image (populate the taskcluster secret? what format should this be in?), and rerun the level 3 image builder task. This may need to wait for :grenade.
3. Generate a new CoT keypair, add it to `scriptworker.constants`, roll out a new scriptworker release + k8s images, then populate the new builder images with the new private key.

It looks like we're doing a combination of (1) and (2). :dhouse rolled back the fingerprint change to unblock Firefox 82.0 RC. And :markco will check with :grenade on how to roll out the images with the existing CoT key. (3) can be a fallback option if we no longer have the existing CoT key. We likely won't be able to fully resolve this issue until tomorrow (October 13), but we have some workarounds.

**[EDIT]** After this is done, we should update the documentation about how to roll out a new hg.m.o fingerprint with these additional steps.

Related: the [artifact metadata RFC](https://github.com/taskcluster/taskcluster-rfcs/pull/158) describes how we can obsolete the Chain of Trust keys, simplifying maintenance.

Currently:

OCC required additional action to deploy `{gecko,comm}-{1,2,3}/b-win2012` images with the new fingerprint. Levels 1 and 2 (untrusted) are now rebuilt and are going green. Level 3 requires a CoT key. :markco doesn't have this key and doesn't know how to create a new image with the trusted CoT key. This means that our trusted level 3 builders are stuck on the old hg.m.o fingerprint, which resulted in the Firefox 82.0 RC graph being stuck.

I saw three potential solutions:

1. Roll back the fingerprint change. Because the new images have both the new and previous fingerprints, they should still work. Because the previous image has the previous fingerprint, those should start working again. However, the fingerprint will expire tomorrow. This is not a good solution after midnight UTC tonight.
2. Find the current CoT private key, figure out how to add it to the image (populate the taskcluster secret? what format should this be in?), and rerun the level 3 image builder task. This may need to wait for :grenade.
3. Generate a new CoT keypair, add it to `scriptworker.constants`, roll out a new scriptworker release + k8s images, then populate the new builder images with the new private key.

It looks like we're doing a combination of (1) and (2). :dhouse rolled back the fingerprint change to unblock Firefox 82.0 RC. And :markco will check with :grenade on how to roll out the images with the existing CoT key. (3) can be a fallback option if we no longer have the existing CoT key. We likely won't be able to fully resolve this issue until tomorrow (October 13), but we have some workarounds.

**[EDIT]** After this is done, we should update the documentation about how to roll out a new hg.m.o fingerprint with these additional steps.

Related: the [artifact metadata RFC](https://github.com/taskcluster/taskcluster-rfcs/pull/158) describes how we can obsolete the Chain of Trust keys, simplifying maintenance.