Closed Bug 1050281 Opened 10 years ago Closed 10 years ago

Clean up / handle case where spot request is active but instance has gone away

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: catlee, Assigned: rail)

References

Details

Bug 1050264 was caused by having active spot requests in AWS without associated instances. We've hit this bug before, and I thought it had been fixed. Perhaps the recent refactoring broke it.

In any case, we should:
a) handle this case in cloudtools.aws.spot.get_available_spot_slave_names by looking at active spot requests and throwing them out if their instances don't exist
b) cancel these requests automatically
This is causing us some pain in us-west-2. rail, welcome back! Could you take a look soonish ?
Severity: normal → major
Flags: needinfo?(rail)
I can look sat this this week, probably by adding another sanity check in http://hg.mozilla.org/build/cloud-tools/file/9dcb80cffe6c/scripts/spot_sanity_check.py.
Assignee: nobody → rail
Flags: needinfo?(rail)
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.