744729 - Sometimes dashboard update script just hangs, preventing further tests until killed

Status: RESOLVED INVALID

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

I haven't diagnosed the exact reason for this, but it is not possible to run further tests if an existing test process has hung. This can cause the dashboard to completely lock up until I manually come in to reset it. I'm not sure exactly why this is: we now time out the actual HDMI capture so as not to fill the disk, so that's not the issue. 

Anyway, regardless of the reason why this is preventing further tests, we just plain shouldn't hang. :) There's a few problems here:

1. Some parts of the various scripts that it calls are susceptible to hangs. For example, we wait indefinitely for the "monkey" script to start here: https://github.com/mozilla/eideticker/blob/master/bin/runtest.py#L79. We should have some kind of timeout. There may be other examples of this as well, need to dig deeper.
2. We really need to just kill everything related to the capture/test after a specified timeout, probably using one of the suggestions here: http://stackoverflow.com/questions/392022/best-way-to-kill-all-child-processes.

Comment 1

[William Lachance (:wlach)]

Haven't seen this in a while, general phone reliability seems to be a larger issue.

Comment 2

[William Lachance (:wlach)]

Haven't seen this in a year, closing.