991935 - host instance: 'None' stopped for 1348 hours.

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jordan Lund (:jlund)]

aws_sanity_checker is reporting an instance with hostname: 'None':

0 None (i-bdb60db4, us-west-2): down for 1348 hours

aws web console surprisingly knows of no such host ;)

I am wary of playing with aws_manage_instances.py to try and terminate it since it holds such a dangerous hostname in python world ;)

My best guess is it does not exist and there is a problem with the sanity checker

Comment 1

[Justin Wood (:Callek)]

Ok, so you can find it in AWS if you go to us-west2 instanced and search by the instance ID

Using that I saw an IP of 10.132.53.241

And then checking DNS against that I find:

$ host 10.132.53.241
Server:  ns1.private.scl3.mozilla.com
Address:  10.22.75.40

Name:    bld-linux64-spot-305.build.releng.usw2.mozilla.com
Address:  10.132.53.241



I find it -really- odd that a spot instance can exist in a stopped state, nevermind in a stopped state with "None"

Comment 2

[Jordan Lund (:jlund)]

there is also:

79 None (i-b30293ba, us-west-2): down for 893 hours

Comment 3

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #2)
> there is also:
> 
> 79 None (i-b30293ba, us-west-2): down for 893 hours

which translates to:

> host 10.132.53.22
22.53.132.10.in-addr.arpa domain name pointer bld-linux64-spot-476.build.releng.usw2.mozilla.com

Comment 4

[Rail Aliiev [:rail]]

I used boto to get the instance object and it looks like something went wrong:

from boto.ec2 import connect_to_region
conn = connect_to_region("us-west-2")
i = conn.get_only_instances(instance_ids=["i-bdb60db4"])[0]
print i.private_ip_address # 10.132.53.241 == bld-linux64-spot-305.build.releng.usw2.mozilla.com.
print i.architecture #i386

# killllll!
i.terminate()