968307 - getting ambigous results for long running instances -> tst-w64-ec2-001 and tst-w64-ec2-002

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jordan Lund (:jlund)]

According to sanity check email these have been up for ~200 hours.

'aws_manage_instances.py status' says they are both running.

Slave Health says they are 'stopped'

AWS web console (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:) can't find them when I search.

slavealloc says they are attached to bm70 for 'Date branch purposes'

I can't ssh or ping them.

I am creating this bug because, as far as I can tell, we are getting conflicting results from various sources. This could be a bug or it could be me failing to understand how things work.

Either way, I don't want to 'aws_manage_instances.py stop tst-w64-ec2-001' until I know more.

Comment 1

[Jordan Lund (:jlund)]

see https://bugzilla.mozilla.org/show_bug.cgi?id=968413#c0

for similar situation regarding slave health vs aws_manage_instances discrepancy

Comment 2

[Rail Aliiev [:rail]]

I think these should be marked as loaned to jhopkins?

Comment 3

[Jordan Lund (:jlund)]

in our sanity email I see ones with 'dev' and ones without in hostname:

== Loaned ==
dev-tst-w64-ec2-002 (i-873c0c8e, us-west-2): Loaned to [e-mail] in an unknown bug, stopped
dev-tst-w64-ec2-001 (i-29566620, us-west-2): Loaned to [e-mail] in an unknown bug, stopped
dev-tst-w64-ec2-000 (i-b21a4c86, us-west-2): Loaned to [e-mail] in an unknown bug, stopped

==== Long running instances ====
0 tst-w64-ec2-002 (i-9a4e93ac, us-west-2): up for 208 hours
1 tst-w64-ec2-001 (i-be06698a, us-west-2): up for 208 hours


The ones that specifically say jhopkins in loaned section, are apparently stopped. I am not sure if they are related but something seems fishy here. Either we have duplicates and aws_manage_instances.py is confused or we have a some instances that are running/hiding in a dark cloud ;)

Comment 4

[John Hopkins (:jhopkins)]

I've terminated the dev-tst-w64-ec2-* instances listed above.

I left the tst-w64-ec2-* instances running.  I see that moz-loaned-to on each of those instances is set to [e-mail] which is fine.