Closed Bug 913541 (panda-0803) Opened 11 years ago Closed 10 years ago

panda-0803 problem tracking

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: bhearsum, Unassigned)

References

Details

12:06 < nagios-releng> Fri 09:06:22 PDT [4822] panda-0803.p9.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_self_test pdu reboot didn't help
(In reply to Ben Hearsum [:bhearsum] from comment #0) > 12:06 < nagios-releng> Fri 09:06:22 PDT [4822] > panda-0803.p9.releng.scl1.mozilla.com is DOWN :CRITICAL: in state > failed_self_test > > > pdu reboot didn't help This belongs in a panda tracking bug, not tegra. :-)
There, I fixed it (thank Ghu, we don't have 803 tegras).
Alias: tegra-0803 → panda-0803
Summary: tegra-0803 problem tracking → panda-0803 problem tracking
recovered by "panda-recovery" bug 902657
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Perma-retrying; disabled in slavealloc.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 959741
Already in ready state. No need for further action.
enabled in slavealloc
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attempting SSH reboot...Failed. Attempting reboot via Mozpool...Failed. Filed IT bug for reboot (bug 996865)
Attempting SSH reboot...Failed. Attempting reboot via Mozpool...Failed. Filed IT bug for reboot (bug 996879)
Status: REOPENED → RESOLVED
Closed: 11 years ago → 10 years ago
Resolution: --- → FIXED
perma-retrying; disabled 03:06:15 INFO - ##### Running request-device step. 03:06:15 INFO - ##### 03:06:15 INFO - Running pre-action listener: _resource_record_pre_action 03:06:15 INFO - Running main action method: request_device 03:06:15 INFO - Getting output from command: ['/builds/panda-0803/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'] 03:06:15 INFO - Copy/paste: /builds/panda-0803/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" 03:06:15 INFO - Reading from file tmpfile_stdout 03:06:15 INFO - Output received: 03:06:15 INFO - /builds/panda-0803/test/build/venv/lib/python2.7/site-packages 03:06:16 INFO - Got request, url=http://mobile-imaging-009.p9.releng.scl1.mozilla.com/api/request/1601157/ 03:06:16 INFO - Waiting for request 'ready' stage. Current state: 'contact_lifeguard' 03:07:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:08:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:09:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:10:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:11:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:12:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:13:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:14:16 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:15:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:16:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:17:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:18:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:19:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:20:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:21:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:22:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:23:17 INFO - Waiting for request 'ready' stage. Current state: 'pending' 03:24:17 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 03:25:17 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 03:26:18 ERROR - INFRA-ERROR: Request did not become ready in time 03:26:18 ERROR - # TBPL EXCEPTION # 03:26:18 INFO - Aborting mozpool request. 03:26:18 INFO - Request 'http://mobile-imaging-009.p9.releng.scl1.mozilla.com/api/request/1601157/' deleted on cleanup 03:26:18 FATAL - Retries limit exceeded 03:26:18 FATAL - Running post_fatal callback... 03:26:18 FATAL - Exiting -1 03:26:18 INFO - Running post-action listener: _resource_record_post_action 03:26:18 INFO - Running post-run listener: _resource_record_post_run 03:26:18 INFO - Running post-run listener: _upload_blobber_files 03:26:18 INFO - Blob upload gear active. 03:26:18 WARNING - Blob upload directory does not exist! program finished with exit code 255 elapsedTime=1255.294764
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
QA Contact: armenzg → bugspam.Callek
Failing and retrying every job.
Depends on: 1027136
This still seems to be taking jobs, even though it is disabled? https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=panda-0803 Last job 20/08/2014 11:19:40
Depends on: 1056143
Due to the problematic nature of this panda, we've decided to decommission it.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → WONTFIX
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.