1016289 - (panda-0110) panda-0110 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Ed Morley [:emorley]]

Perma-retrying; disabled.

21:11:38     INFO - #####
21:11:38     INFO - ##### Running request-device step.
21:11:38     INFO - #####
21:11:38     INFO - Running pre-action listener: _resource_record_pre_action
21:11:38     INFO - Running main action method: request_device
21:11:38     INFO - Getting output from command: ['/builds/panda-0110/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())']
21:11:38     INFO - Copy/paste: /builds/panda-0110/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
21:11:38     INFO - Reading from file tmpfile_stdout
21:11:38     INFO - Output received:
21:11:38     INFO -  /builds/panda-0110/test/build/venv/lib/python2.7/site-packages
21:11:38     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
21:11:38     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
21:11:39     INFO - Got request, url=http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/1408885/
21:11:39     INFO - Waiting for request 'ready' stage.  Current state: 'contact_lifeguard'
21:12:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:13:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:14:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:15:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:16:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:17:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:18:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:19:39     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:20:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:21:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:22:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:23:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:24:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:25:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:26:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:27:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:28:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:29:40     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
21:30:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
21:31:40    ERROR - INFRA-ERROR: Request did not become ready in time
21:31:40    ERROR - # TBPL EXCEPTION #
21:31:40     INFO - Aborting mozpool request.
21:31:41     INFO - Request 'http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/1408885/' deleted on cleanup
21:31:41    FATAL - Retries limit exceeded
21:31:41    FATAL - Running post_fatal callback...
21:31:41    FATAL - Exiting -1
21:31:41     INFO - Running post-action listener: _resource_record_post_action
21:31:41     INFO - Running post-run listener: _resource_record_post_run
21:31:41     INFO - Running post-run listener: _upload_blobber_files
21:31:41     INFO - Blob upload gear active.
21:31:41  WARNING - Blob upload directory does not exist!

Comment 1

[Vinh Hua [:vinh]]

SD card replaced.  Panda is now in ready state.

Comment 2

[Chris Cooper [:coop] (he/him)]

Re-enabled in slavealloc.

Comment 3

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 1113009)

Comment 4

[Van Le [:van]]

replaced preseeded SD card, panda passed self test.

Comment 5

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 1114855)

Comment 6

[Van Le [:van]]

replaced SD card with preseed image, panda passed self test.

Comment 7

[Chris Cooper [:coop] (he/him)]

Killed off a buildbot process from Dec 13.

Comment 8

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 1127994)

Comment 9

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 1132593)