1610553 - WorkerPoolId not set for some AWS workers

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Chris Cooper [:coop] (he/him)]

Looking at the AWS month-to-date numbers for January, I notice some instances that are still not tagged with WorkerPoolId. Here are the instance types:

• g2.2xlarge - $9,358.98; 46,843.04 hrs
• c5.2xlarge - $2,542.61; 18,047.53 hrs
• c4.2xlarge - $1,776.83; 14,267.29 hrs
• a bunch more with <$1K and/or <1K hrs

ref: Cost Explorer query

Further digging maps the instances as follows:

• g2.2xlarge -> gecko-t/t-win7-32-gpu
• c5.2xlarge -> gecko-t/t-win10-64
• c4.2xlarge -> gecko-t/t-win7-32

Ideally all instance s should be tagged with WorkerPoolId, but let's start with these big omissions first.

Comment 1

[Chris Cooper [:coop] (he/him)]

Here's the long tail, modulo $3 worth of t2 instance usage:

• c5d.4xlarge - Name:windows-ami-builder (US West Oregon) $732.45, 488 hrs, Box
• g3.4xlarge - WorkerType:golden-gecko-t-win10-64-gpu-b (US West Oregon) $555.18, 488 hrs, Box (No Name tag)
• m4.16xlarge - Name:gecko-1-b-win2012-xlarge (EU Frankfurt) $435.98, 488 hrs, Spot
• m5.4xlarge - Name:gecko-3/b-linux (US East N. Virginia) $263.95, 809.28 hrs, Spot
• c4.4xlarge - Name:gecko-[1,3]/b-win2012 (US East N. Virginia) $244.35, 974 hrs, Spot
• c5.4xlarge - Name:gecko-1/b-win2012 (US East N. Virginia) $127.38, 486 hrs, Spot
• m5.large - Name:gecko-t/t-linux-large (US West Oregon, US East N. Virginia) $49.83, 1463.3 hrs, Spot
• c3.xlarge - Name:gecko-t/t-linux-xlarge (US West Oregon) $29.13, 486 hr, Spot

Two patterns that leap out at me:

1. Windows workers of all types seem to be missing WorkerPoolId
2. Some subset of linux workers (gecko-3/b-linux, gecko-t/t-linux-(x)large) are also missing WorkerPoolId

Comment 2

[Chris Cooper [:coop] (he/him)]

:dustin did a little more digging, and all the instances he could find had been spawned between Nov 11 and Dec 10 and were still running. It's unclear whether they were still taking jobs, but that's a separate issue.

EC2 console query for running instances with no WorkerPoolId tag

I'm currently iterating through regions killing off these older instances and then will report on what's left at the end.

Comment 3

[Chris Cooper [:coop] (he/him)]

Cleanup is done.

In all, I terminated between 150-200 instances (mostly Windows) that were created before Dec 10. The Dec 10 date is key, because on that day :bstack released v24.1.0 of Taskcluster to deal with the tagging issue and help catch zombie instances.

Conveniently, this probably also fixes the "over-provisioning" issue that :catlee had reported for Windows instances. Having 150-200 Windows instances running but never taking work can't be good for the efficiency stats.

Leaving this open with a NI for myself to file a follow-up tomorrow for better automated reporting/culling of zombies.

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] pronoun: he from comment #3)

Leaving this open with a NI for myself to file a follow-up tomorrow for better automated reporting/culling of zombies.

Worker-manager does a better job of this now for properly-tagged instances. Filed bug 1613224 to make sure we have some back-stop reporting for any instances that get missed.