888577 - (panda-0834) panda-0834 problem tracking

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Ed Morley [:emorley]]

This panda has failed at least the last 25 jobs:
https://secure.pub.build.mozilla.org/buildapi/recent/panda-0834

https://tbpl.mozilla.org/php/getParsedLog.php?id=24673069&tree=Cedar

{
13:17:40     INFO - #####
13:17:40     INFO - ##### Running request-device step.
13:17:40     INFO - #####
13:17:40     INFO - Getting output from command: ['/builds/panda-0834/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())']
13:17:40     INFO - Copy/paste: /builds/panda-0834/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
13:17:40     INFO - Reading from file tmpfile_stdout
13:17:40     INFO - Output received:
13:17:40     INFO -  /builds/panda-0834/test/build/venv/lib/python2.7/site-packages
13:17:40     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
13:17:40     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
13:17:41     INFO - Got request, url=http://mobile-imaging-009.p9.releng.scl1.mozilla.com/api/request/212369/
13:17:41     INFO - Waiting for request 'ready' stage.  Current state: 'contacting_lifeguard'
13:18:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:19:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:20:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:21:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:22:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:23:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:24:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:25:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:26:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:27:41     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:28:42     INFO - Waiting for request 'ready' stage.  Current state: 'pending'
13:29:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:30:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:31:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:32:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:33:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:34:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:35:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:36:42     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
13:37:42    ERROR - INFRA-ERROR: Request did not become ready in time
}

Comment 1

[Ed Morley [:emorley]]

Please can this panda be disabled :-)

Comment 2

[Justin Wood (:Callek)]

[foopy98.build.mozilla.org] run: echo "disable please" > ./disabled.flg
[OK]   Stopped panda-0834

Kim, let's touch base on steps to recover pandas in this state before you go on your trip. I'm sure it's easy I just don't know how

Comment 3

[Ed Morley [:emorley]]

Thank you :-)

Comment 4

[Kim Moir [:kmoir] ET]

I don't know what to do when the pandas get in this state.  Sometimes this seems to be an intermittent error.  Buildbot assigns a slave for the build and when mozpool verifies it it doesn't get into a ready state within the timeouts specified in mozpool.  However, this reminds me that I should update the panda documentation for builduty to describe how deal with mozpool now that the non-talos tests are using it.

Comment 5

[Jake Watkins [:dividehex]]

I didn't see any obvious issue with this panda board.  Re-imaged with Android via mozpool.  If this continues to be problem prone panda, SD card replacement is the next step.

Comment 6

[Justin Wood (:Callek)]

Resolving all panda bugs linked from Bug 817103 that are not in troubleshoot or failed_pxe* state in lifeguard.

Comment 7

[Justin Wood (:Callek)]

recovered by "panda-recovery" bug 902657

Comment 8

[Ed Morley [:emorley]]

Perma-retrying - disabled.

Comment 9

[Chris Cooper [:coop] (he/him)]

Due to the problematic nature of this panda, we've decided to decommission it.