817057 - mozpool should poll "free" devices (and maybe "ready") periodically

Status: RESOLVED FIXED

(Testing Graveyard :: Mozpool, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We took this out earlier because it was rebooting pandas that other teams were using outside of mozpool.  With the 'locked-out' state in bug 817035 and plans to eventually manage ALL THE DEVICES with mozpool, that's less of a problem now.

We want to use the device state from mozpool as a monitoring tool, too, but that requires that it be reasonably up to date.  Especially for "free" devices, we'd like to know
 * if it stops pinging
 * if it stops responding to SUT requests
 * (maybe) if its CPU is hot (polled via SUT)

My thinking is that the "free" state will loop, and on each loop it will kick off its various checks.  If any return a negative result, it will attempt a power-cycle, in hopes that that will fix it, or lead to a failed_pc_pinging state if the device is truly dead.  A failure of the temp check should probably lead immediately to "failed_over_temp", powering the device off on entry.

I'd like to do this in the "ready" (i.e., busy satisfying a mozpool request) state, too, but I'm not sure if that would interfere with running tests.  On the other hand, I find it unlikely that devices are overheating in the "free" (idle) state.  I'm open to thoughts on this.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug817057.patch

Comment 2

[Jake Watkins [:dividehex]]

lgtm.  I think polling for temp while in a "free" state is where we will get the most telling data if a panda is overheating in that state.

Comment 3

[Mark Côté [:mcote]]

Comment on attachment 696433 [details] [diff] [review]
bug817057.patch

Review of attachment 696433 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.