1594369 - deepspeech win-b tasks pending piling up

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[:gerard-majax]

We have 20 pending task for windows instances: https://community-tc.services.mozilla.com/worker-manager

There is no meaningful error reported: https://community-tc.services.mozilla.com/worker-manager/proj-deepspeech%2Fwin-b/errors

Comment 1

[Pete Moore [:pmoore][:pete]]

It looks at first glance like instances aren't being spawned, rather than that they are being spawned but not taking jobs:

https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Instances:tag:Name=proj-deepspeech/win-b;sort=tag:Name
https://us-west-1.console.aws.amazon.com/ec2/home?region=us-west-1#Instances:tag:Name=proj-deepspeech/win-b;sort=tag:Name
https://us-west-2.console.aws.amazon.com/ec2/home?region=us-west-2#Instances:tag:Name=proj-deepspeech/win-b;sort=tag:Name

Comment 2

[Pete Moore [:pmoore][:pete]]

My bad, it turns out I was looking in the wrong amazon account...

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

These are in a different AWS account than where Pete was looking. Instances are being spawned, and in fact there are 16 of them, but they are probably crashing out of generic-worker but not shutting down. I've given Pete access to the right AWS account (sorry!) and he's going to have a look-see.

Comment 4

[Pete Moore [:pmoore][:pete]]

The problem was malformed secrets. These were deleted, old workers were terminated, and things returned to normal.