Closed Bug 1287921 Opened 8 years ago Closed 8 years ago

aws_stop_idle can't stop tst-linux32 instances

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: catlee, Unassigned)

Details

I noticed this afternoon that we have a lot of tst-linux32 instances running. Looking at aws_stop_idle logs shows that it can't log into the machines to check if they're idle:

2016-07-19 13:32:56,965 - DEBUG -  Couldn't log into tst-linux32-spot-1043 at 10.134.40.158
2016-07-19 13:32:57,468 - DEBUG -  Couldn't log into tst-linux32-spot-310 at 10.132.46.216
2016-07-19 13:32:58,737 - DEBUG -  Couldn't log into tst-linux32-spot-590 at 10.132.40.35
2016-07-19 13:32:59,209 - DEBUG -  Couldn't log into tst-linux32-spot-551 at 10.132.156.106
2016-07-19 13:33:00,457 - DEBUG -  Couldn't log into tst-linux32-spot-729 at 10.134.157.109
2016-07-19 13:33:00,493 - DEBUG -  Couldn't log into tst-linux32-spot-360 at 10.132.46.121
2016-07-19 13:33:05,435 - DEBUG -  Couldn't log into tst-linux32-spot-393 at 10.132.46.134
2016-07-19 13:33:05,616 - DEBUG -  Couldn't log into tst-linux32-spot-543 at 10.132.47.33
2016-07-19 13:33:08,409 - DEBUG -  Couldn't log into tst-linux32-spot-394 at 10.132.40.110
2016-07-19 13:33:09,156 - DEBUG -  Couldn't log into tst-linux32-spot-1002 at 10.134.59.251
2016-07-19 13:33:09,806 - DEBUG -  Couldn't log into tst-linux32-spot-237 at 10.134.156.15
2016-07-19 13:33:10,770 - DEBUG -  Couldn't log into tst-linux32-spot-1038 at 10.134.156.229
2016-07-19 13:33:12,510 - DEBUG -  Couldn't log into tst-linux32-spot-712 at 10.134.157.181
As :catlee spotted, the linux32 golden AMI was running since May so it got terminated and regenerated. At the moment, the number of running instances has dropped and they're all launched using the fresh AMI. 
It turns out that the old AMI didn't include the changes done for releng private keys rotation (from aws-manager, I was able to manually connect to a new instance, but not to an older one).
Things look fine, marking this as resolved for now.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.