Closed Bug 1307894 Opened 3 years ago Closed 2 months ago

[SpotManager] Sometimes does not complete

Categories

(Testing :: ActiveData, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ekyle, Assigned: ekyle)

Details

For some reason the SpotManager does not shutdown...

> ip-172-31-0-63 - 2016-10-03 09:20:54 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:20:54 - lifecycle watcher - "spot_manager.py:456" (life_cycle_watcher) - waiting for spot requests: ["sir-ctdga5dn"]
> ip-172-31-0-63 - 2016-10-03 09:20:59 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:21:04 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-03 09:21:05 - lifecycle watcher - "etl.py:50" (setup) - setup i-effbe232
> ip-172-31-0-63 - 2016-10-03 09:21:09 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:31 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:36 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:41 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"
> ip-172-31-0-63 - 2016-10-05 19:01:46 - Main Thread - "threads.py:533" (join) - "Main Thread" waiting on thread "lifecycle watcher"

It appears the "lifecycle watcher" begins setup of a machine, but never gets to the next step. 

This is the second time it has happened in the past month, or two. It prevents further machine bidding; eventually leading to no machines.

Not been a problem for a while

Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.