Closed Bug 1715749 Opened 5 years ago Closed 5 years ago

rotate cot keys on gecko-3/b-win2012

Tracking

(Not tracked)

Status:

RESOLVED FIXED

People

(Reporter: grenade, Assigned: grenade)

Details

Attachments

(14 files, 1 obsolete file)

Bug 1715749 - new 2012 images with rotated cot keys r=aki 5 years ago Rob Thijssen [:grenade (EET/UTC+0300)] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1715749 - Backed out changeset 9860b1d898ec r=jcristau 5 years ago Mihai Tabara [:mtabara]⌚️GMT 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1715749 - 2nd try new 2012 images with rotated cot keys r=aki 5 years ago Mark Cornmesser [:markco] 48 bytes, text/x-phabricator-request		Details \| Review
standalone generic-worker repo log from v16.5.1 until migration 5 years ago Aki Sasaki (not active) 19.94 KB, text/plain		Details
standalone generic-worker repo diff from v16.5.1 until migration 5 years ago Aki Sasaki (not active) 1.00 MB, text/plain		Details
monorepo generic-worker directory log from migration until now 5 years ago Aki Sasaki (not active) 42.44 KB, text/plain		Details
monorepo generic-worker directory diff from migration until now 5 years ago Aki Sasaki (not active) 953.28 KB, text/plain		Details
standalone taskcluster-proxy repo log from the v16.5.1 timeframe until migration 5 years ago Aki Sasaki (not active) 3.64 KB, text/plain		Details
standalone taskcluster-proxy repo diff from the v16.5.1 timeframe until migration 5 years ago Aki Sasaki (not active) 48.07 KB, patch		Details \| Diff \| Splinter Review
monorepo taskcluster-proxy directory log from migration until now 5 years ago Aki Sasaki (not active) 6.90 KB, text/plain		Details
Bug 1715749 - tc-tools 43.2 for b-win2012 r=aki 5 years ago Rob Thijssen [:grenade (EET/UTC+0300)] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1715749 - generic-worker and capacity changes for new ami. r=#releng 5 years ago Aki Sasaki (not active) 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1715749 - win: set deploymentId, omit livelogPUTPort r=aki 5 years ago Rob Thijssen [:grenade (EET/UTC+0300)] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1715749 - win: correction to wstServerURL r=aki 5 years ago Rob Thijssen [:grenade (EET/UTC+0300)] 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1715749 - populate deploymentId in AWS generic workers if not set. r=#releng 5 years ago Aki Sasaki (not active) 48 bytes, text/x-phabricator-request		Details \| Review

Rob Thijssen [:grenade (EET/UTC+0300)]

Assignee

Description

•

5 years ago

•

Edited

i documented today's rotation at: https://gist.github.com/grenade/7d5c9f567951fbc3c3caf023bd356fa6 and have pasted the same below.

windows cot key rotation

trigger occ ami builds

git clone https://github.com/mozilla-releng/OpenCloudConfig ./occ
cd ./occ
git commit -m "cot key rotation $(date --iso-8601)" -m "deploy: gecko-3-b-win2012" --allow-empty

deploy key to image build instance
1. obtain instance credentials from ec2 userdata
  - option 1: using the console:
    - navigate to the "edit userdata" page of the running instance (see figure 01 below)
    - look for a line containing the administrator password. eg:
```
<rootPassword>xxx999!!!</rootPassword>
```
  - option 2: using the aws cli:
```
# assuming a `[moz-occ]` profile exists in ~/.aws/credentials
instance_id=$(aws ec2 describe-instances \
  --profile moz-occ \
  --region us-west-2 \
  --filters Name=key-name,Values=mozilla-taskcluster-worker-gecko-3-b-win2012 Name=instance-state-name,Values=running \
  --query Reservations[].Instances[].InstanceId \
  --output text)
instance_password=$(aws ec2 describe-instance-attribute \
  --profile moz-occ \
  --region us-west-2 \
  --instance-id ${instance_id} \
  --attribute userData \
  --query UserData.Value \
  --output text | base64 --decode | grep rootPassword | sed -e 's/\s*<[^>]*>//g')
```
2. connect to the instance over remote desktop (rdp). note that the base windows 2012 image does not support tls 1.2 connections before the instance is bootstrapped completely. this means your rdp client will fail to connect unless it supports tls 1.0
```
instance_ip=$(aws ec2 describe-instances \
  --profile moz-occ \
  --region us-west-2 \
  --filters Name=key-name,Values=mozilla-taskcluster-worker-gecko-3-b-win2012 Name=instance-state-name,Values=running \
  --query Reservations[].Instances[].PublicIpAddress \
  --output text)
xfreerdp \
  /u:Administrator \
  /p:"${instance_password}" \
  /cert-ignore \
  /kbd:809 \
  /w:2400 \
  /h:1200 \
  +clipboard \
  /v:${instance_ip}
```
3. open a text editor (sublime text is installed and uses sane file encodings by default) and paste the ed25519 cot private key into a new file. no newline is required after the key. save the file to C:\generic-worker\ed25519-private.key
4. log out from the rdp session or shut down the instance (do not just close the connection. this will leave the occ session hanging and the image build incomplete).
5. confirm that the occ image build task completes successfully. after the instance shuts down, occ will build an ami from it and distribute that ami to several regions. this can take half an hour. you can get to the image build task by clicking on the ci build associated with your commit and push from step 1 on the occ commits page. the task of interest is named Update gecko-3-b-win2012 AMIs and should complete green
6. to troubleshoot an incomplete image build instance, examine the instance logs at: https://my.papertrailapp.com/events?q=system%3A${instance_id}
7. to see ami availability look in the ec2 console:
update ci configuration with the new ami ids obtained from the artifacts page of the task described in 2.v above. the artifact of interest is named ami-latest.json / ami-latest.yml.

Rob Thijssen [:grenade (EET/UTC+0300)]

Assignee

Comment 1

•

5 years ago

Attached file Bug 1715749 - new 2012 images with rotated cot keys r=aki — Details

Phabricator Automation

Updated

•

5 years ago

Assignee: nobody → rthijssen

Status: NEW → ASSIGNED

Pulsebot

Comment 2

•

5 years ago

Pushed by asasaki@mozilla.com: https://hg.mozilla.org/ci/ci-configuration/rev/9860b1d898ec new 2012 images with rotated cot keys r=aki

Mihai Tabara [:mtabara]⌚️GMT

Comment 3

•

5 years ago

Attached file Bug 1715749 - Backed out changeset 9860b1d898ec r=jcristau — Details

Pulsebot

Comment 4

•

5 years ago

Pushed by mtabara@mozilla.com: https://hg.mozilla.org/ci/ci-configuration/rev/1a7003304c47 Backed out changeset 9860b1d898ec r=jcristau

Mihai Tabara [:mtabara]⌚️GMT

Comment 5

•

5 years ago

Backing this out as all workers with the new AMI seem to die before claiming any task. The only green ones are the ones with the old AMI, so I'm backing out the patch until we understand what's going on.

Rob Thijssen [:grenade (EET/UTC+0300)]

Assignee

Comment 6

•

5 years ago

the problem was caused by some no longer supported (by generic-worker/worker-runner) worker-pool configuration settings. specifically: capacity and livelogPUTPort (at: https://firefox-ci-tc.services.mozilla.com/worker-manager/gecko-3%2Fb-win2012).

the key rotation change also updates generic worker (and other tc-tools) to 43.2.0 (from 16.5.1), however:

43.2.0 will not claim tasks if capacity and livelogPUTPort are set.
16.5.1 uses those settings.

i had also noticed the failure to take tasks and had already corrected pool configuration to work correctly for 43.2.0 but my changes overlapped the rollback to old amis and 16.5.1.

i am on pto, so i'll just revert my changes as well so that the pool config contains capacity and livelogPUTPort again.

if there is appetite to get the key rotation and tc-tools update back in, whoever does the ci-config merge will also need to modify the resulting worker pool configuration at: https://firefox-ci-tc.services.mozilla.com/worker-manager/gecko-3%2Fb-win2012 so that each launchConfig does mot contain capacity and livelogPUTPort.

additionally to support 43.2.0, the genericWorker.config block for each launchConfig should contain:

"disableReboots": false, (not "disableReboots": true, as required by 16.5.1)
"shutdownMachineOnIdle": true, (not "shutdownMachineOnIdle": false, as required by 16.5.1)

if someone wants to reintroduce the key rotation and tc-tools update today, i am happy to assist with these changes but i would prefer if this didn't drag on into the coming days.

Mihai Tabara [:mtabara]⌚️GMT

Comment 7

•

5 years ago

•

Edited

Bah, seems like we race-conditioned there, too bad. I'm around today to help with re-landing this or we can coordinate on (re)deploying this when you're back from PTO.

Rob Thijssen [:grenade (EET/UTC+0300)]

Assignee

Comment 8

•

5 years ago

i did not revert my config changes but i see that the old amis seem to work fine with the new settings (no capacity and livelogPUTPort settings, "disableReboots": false,, "shutdownMachineOnIdle": true,). all pending tasks have been claimed. so, i will leave settings as they are since they don't seem to affect 16.5.1 and we must have them for 43.2.0.