getting ambigous results for long running instances -> tst-w64-ec2-001 and tst-w64-ec2-002

RESOLVED FIXED

Status

RESOLVED FIXED
5 years ago
2 years ago

People

(Reporter: jlund, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
According to sanity check email these have been up for ~200 hours.

'aws_manage_instances.py status' says they are both running.

Slave Health says they are 'stopped'

AWS web console (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:) can't find them when I search.

slavealloc says they are attached to bm70 for 'Date branch purposes'

I can't ssh or ping them.

I am creating this bug because, as far as I can tell, we are getting conflicting results from various sources. This could be a bug or it could be me failing to understand how things work.

Either way, I don't want to 'aws_manage_instances.py stop tst-w64-ec2-001' until I know more.
(Reporter)

Comment 1

5 years ago
see https://bugzilla.mozilla.org/show_bug.cgi?id=968413#c0

for similar situation regarding slave health vs aws_manage_instances discrepancy
I think these should be marked as loaned to jhopkins?
Flags: needinfo?(jhopkins)
(Reporter)

Comment 3

5 years ago
in our sanity email I see ones with 'dev' and ones without in hostname:

== Loaned ==
dev-tst-w64-ec2-002 (i-873c0c8e, us-west-2): Loaned to jhopkins@mozilla.com in an unknown bug, stopped
dev-tst-w64-ec2-001 (i-29566620, us-west-2): Loaned to jhopkins@mozilla.com in an unknown bug, stopped
dev-tst-w64-ec2-000 (i-b21a4c86, us-west-2): Loaned to jhopkins@mozilla.com in an unknown bug, stopped

==== Long running instances ====
0 tst-w64-ec2-002 (i-9a4e93ac, us-west-2): up for 208 hours
1 tst-w64-ec2-001 (i-be06698a, us-west-2): up for 208 hours


The ones that specifically say jhopkins in loaned section, are apparently stopped. I am not sure if they are related but something seems fishy here. Either we have duplicates and aws_manage_instances.py is confused or we have a some instances that are running/hiding in a dark cloud ;)
I've terminated the dev-tst-w64-ec2-* instances listed above.

I left the tst-w64-ec2-* instances running.  I see that moz-loaned-to on each of those instances is set to jhopkins@mozilla.com which is fine.
Flags: needinfo?(jhopkins)
(Reporter)

Updated

5 years ago
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.