1307894 - [SpotManager] Sometimes does not complete

Status: RESOLVED FIXED

(Testing Graveyard :: ActiveData, defect)

Description

[Kyle Lahnakoski [:ekyle]]

For some reason the SpotManager does not shutdown...

> ip-172-31-0-63 - 2016-10-03 09:20:54 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:20:54 - lifecycle watcher - "spot_manager.py:456" (life_cycle_watcher) - waiting for spot requests: ["sir-ctdga5dn"]
> ip-172-31-0-63 - 2016-10-03 09:20:59 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:21:04 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:21:05 - lifecycle watcher - "etl.py:50" (setup) - setup i-effbe232
> ip-172-31-0-63 - 2016-10-03 09:21:09 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:31 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:36 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:41 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:46 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"

It appears the "lifecycle watcher" begins setup of a machine, but never gets to the next step. 

This is the second time it has happened in the past month, or two. It prevents further machine bidding; eventually leading to no machines.

Comment 1

[Kyle Lahnakoski [:ekyle]]

Not been a problem for a while