[SpotManager] Sometimes does not complete

NEW
Assigned to

Status

Testing
ActiveData
11 months ago
11 months ago

People

(Reporter: ekyle, Assigned: ekyle)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

11 months ago
For some reason the SpotManager does not shutdown...

> ip-172-31-0-63 - 2016-10-03 09:20:54 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:20:54 - lifecycle watcher - "spot_manager.py:456" (life_cycle_watcher) - waiting for spot requests: ["sir-ctdga5dn"]
> ip-172-31-0-63 - 2016-10-03 09:20:59 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:21:04 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:21:05 - lifecycle watcher - "etl.py:50" (setup) - setup i-effbe232
> ip-172-31-0-63 - 2016-10-03 09:21:09 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:31 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:36 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:41 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:46 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"

It appears the "lifecycle watcher" begins setup of a machine, but never gets to the next step. 

This is the second time it has happened in the past month, or two. It prevents further machine bidding; eventually leading to no machines.
You need to log in before you can comment on or make changes to this bug.