Bug 1050264 was caused by having active spot requests in AWS without associated instances. We've hit this bug before, and I thought it had been fixed. Perhaps the recent refactoring broke it. In any case, we should: a) handle this case in cloudtools.aws.spot.get_available_spot_slave_names by looking at active spot requests and throwing them out if their instances don't exist b) cancel these requests automatically
This is causing us some pain in us-west-2. rail, welcome back! Could you take a look soonish ?
I can look sat this this week, probably by adding another sanity check in http://hg.mozilla.org/build/cloud-tools/file/9dcb80cffe6c/scripts/spot_sanity_check.py.