900857 - (panda-0770) panda-0770 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Ed Morley [:emorley]]

Failed all of the last 100 jobs:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=panda-0770

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=26057167&tree=Fx-Team

{
21:31:16     INFO - #####
21:31:16     INFO - ##### Running request-device step.
21:31:16     INFO - #####
21:31:16     INFO - Running pre-action listener: _resource_record_pre_action
21:31:16     INFO - Running main action method: request_device
21:31:16     INFO - Getting output from command: ['/builds/panda-0770/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())']
21:31:16     INFO - Copy/paste: /builds/panda-0770/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
21:31:16     INFO - Reading from file tmpfile_stdout
21:31:16     INFO - Output received:
21:31:16     INFO -  /builds/panda-0770/test/build/venv/lib/python2.7/site-packages
21:31:16     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
21:31:16     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
21:31:16     INFO - Got request, url=http://mobile-imaging-008.p8.releng.scl1.mozilla.com/api/request/284388/
21:31:17     INFO - Waiting for request 'ready' stage.  Current state: 'finding_device'
21:32:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:33:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:34:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:35:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:36:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:37:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:38:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:39:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:40:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:41:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:42:17     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:43:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:44:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:45:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:46:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:47:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:48:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:49:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:50:18     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
21:51:18    ERROR - INFRA-ERROR: Request did not become ready in time
21:51:18    ERROR - # TBPL EXCEPTION #
21:51:18    FATAL - Retries limit exceeded
}

Please can this be disabled.

Comment 1

[John Hopkins (:jhopkins)]

Forced state "locked_out" via mozpool.

Comment 2

[John Hopkins (:jhopkins)]

Changed state to "disabled" per dustin

Comment 3

[John Hopkins (:jhopkins)]

Disabled via manage_foopies.py.  Reset state to "ready" and ran self-test via lifeguard/mozpool.

Comment 4

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 5

[Justin Wood (:Callek)]

recovered by "panda-recovery" bug 902657