893511 - Trees closed - Panda test slaves are all failing with "INFRA-ERROR: Request did not become ready in time"

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

On inbound (where jobs are actually trying to run), both on the retriggered build on the push from before the tree closure and on the one push after the trees were reopened, every Panda/Android 4.0 test is failing like

https://tbpl.mozilla.org/php/getParsedLog.php?id=25268671&tree=Mozilla-Inbound

20:34:06     INFO - #####
20:34:06     INFO - ##### Running request-device step.
20:34:06     INFO - #####
20:34:07     INFO - Getting output from command: ['/builds/panda-0731/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())']
20:34:07     INFO - Copy/paste: /builds/panda-0731/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
20:34:07     INFO - Reading from file tmpfile_stdout
20:34:07     INFO - Output received:
20:34:07     INFO -  /builds/panda-0731/test/build/venv/lib/python2.7/site-packages
20:34:07     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
20:34:07     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
20:34:07     INFO - Got request, url=http://mobile-imaging-008.p8.releng.scl1.mozilla.com/api/request/239790/
20:34:07     INFO - Waiting for request 'ready' stage.  Current state: 'finding_device'
20:35:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:36:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:37:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:38:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:39:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:40:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:41:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:42:07     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:43:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:44:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:45:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:46:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:47:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:48:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:49:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:50:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:51:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:52:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:53:08     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
20:54:08    ERROR - INFRA-ERROR: Request did not become ready in time
20:54:08    ERROR - # TBPL EXCEPTION #
20:54:08    FATAL - Retries limit exceeded
20:54:08    FATAL - Exiting -1
program finished with exit code 255

Closed  fx-team, mozilla-aurora, mozilla-beta, mozilla-central, mozilla-inbound, mozilla-release, services-central (which I think is everything that runs them).

Comment 1

[Jake Watkins [:dividehex]]

mozpool daemon on mobile-imaging-008 and 009 got stuck on a hung thread again.  Both daemons were restarted and device states are moving now.  We should probably land the patches to handle this sooner rather than later.  :-}

device WARNING - [2013-07-14 01:12:50,519] polling thread still running at 45827s; not starting another

See https://bugzilla.mozilla.org/show_bug.cgi?id=817762

Comment 2

[Jake Watkins [:dividehex]]

After checking all the mobile-imaging servers, we can add 007 and 010 to that list.  mozpool restarted on both.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

We look to be in a normal state since the imaging server fixes, lots of green builds and only a smalle proportion retries. Please reopen if there are issues that were missed.