843744 - (panda-0081) Decommission panda-0081

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task, P3)

Description

[Kim Moir [:kmoir] ET]

DC ops is looking at this panda, as it's very flaky.

See bug 817103 for more details

Comment 1

[Kim Moir [:kmoir] ET]

Also forgot to mention that I chmoded og-r the panda-0081 directory on foopy85 so it would stay out of commission

Comment 2

[Kim Moir [:kmoir] ET]

 echo "bug 843744" > /builds/panda-0081/disabled.flg is a better way to disable a panda

Comment 3

[Jake Watkins [:dividehex]]

The issue that first caused this panda to fail (5 months ago) might have been an issue with the automation process and not really a problem with the panda board itself.  Since I really didn't find anything wrong with this panda board, I issued a reimage via mozpool.

Comment 4

[Justin Wood (:Callek)]

Resolving all panda bugs linked from Bug 817103 that are not in troubleshoot or failed_pxe* state in lifeguard.

Comment 5

[[e-mail] (:bhearsum)]

pdu reboot didn't help, needs recovery

Comment 6

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 7

[Vinh Hua [:vinh]]

panda-081 -  selftest.py[INFO]: test_preseed_file_integrity[FAILED] boot.scr : 5a5c34aa07d2d8f23e1b69347d49bacf205041dd != 6261fdd19a45db13e6503c5010e3917dbb13eeed

fix - replaced SD card with correct preseed.

Comment 8

[[e-mail] (:bhearsum)]

Back in production

Comment 9

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 1067672)

Comment 10

[Phil Ringnalda (:philor)]

Didn't recover, hasn't taken a job for 26 days.

Comment 11

[Chris Cooper [:coop] (he/him)]

Panda was added to the proper foopy and is now taking jobs.

Comment 12

[Phil Ringnalda (:philor)]

Failing every other job, disabled in slavealloc.

Comment 13

[Chris Cooper [:coop] (he/him)]

Tools repo updated on foopy so that this panda can be properly rebooted again.

Comment 14

[Phil Ringnalda (:philor)]

Since then, 2 green, 1 orange, 17 retry, 0 jobs for the last 4 days. Not sure this was a good choice for backfill. Disabled.

Comment 15

[Van Le [:van]]

replaced SD card, panda passed self test

Comment 16

[Phil Ringnalda (:philor)]

Reenabled and rebooted.

Comment 17

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 1082738)

Comment 18

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 1083006)

Comment 19

[Van Le [:van]]

replaced SD card, panda passed self test.

panda-0081	ready (request 2313518)

Comment 20

[Phil Ringnalda (:philor)]

Score for the 2014-10-10 SD card: 11 retries, 2 failures, 5 passes.
Score for the 2014-10-20 SD card: 3 retries, 2 failures, 0 passes (and I'm already calling that a complete score since it hasn't taken a job for three days now).

It ain't the SD card. Disabled in slavealloc so we can skip another pointless card replacement the next time it fails to reboot.

Comment 21

[Chris Cooper [:coop] (he/him)]

Let's decomm this one.

Comment 22

[Vinh Hua [:vinh]]

Hey MOC team can you remove this host from nagios before I physically decomm it?  Thanks.

Comment 23

[Ludovic Hirlimann [:Usul]]

(In reply to Vinh Hua [:vinh] from comment #22)
> Hey MOC team can you remove this host from nagios before I physically decomm
> it?  Thanks.

Panda's are commented in nagios. Can you update the status of the bug ?

Comment 24

[Vinh Hua [:vinh]]

Panda-0081 has been physically decomm'd.